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I.  INTRODUCTION 


Tim;  der.oribcra  results  of  the  second  year's  work  on  an  analytic 

proa  ms  node!  (APM)  for  systems  design  and  measurement.  The  work  was 
completed  by  Dunlap  and  Associates,  Inc.,  under  contract  MDA903-80-C-03154 
between  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Meier  ces  (Fort  Benning  Field  Unit),  and  the  Mellonics  Systems  Development 
r>iv<  ion  ul  I.itton  Systems,  Inc.  Task  3  (System  Development  and  Evaluation 
Ter  honing  y )  of  that  contract  was  performed  by  Dunlap  under  a  subcontract 
f No.  UatT’H)  from  Litton  Mellonics. 

During  the  second  year  this  task  has  been  directed  primarily  toward  I 
L-mon  stealing  that  the  APM,  as  originally  designed  for  systems  measurement  , 
could  be  applied  successfully  to  the  analysis  and  evaluation  of  training 
•system*.  This  liar,  been  clone  in  the  context  of  the  Bradley  Infantry  Fighting 
•/chicle  Training  System  (BIFVTS).  The  project  staff  also  explored  the  use  of 
be  APM,  anr!  its  component  systems  taxonomy  model  (STM),  as  a  design  tool 
'or  tie  specification  of  training  systems  required  to  support  developing 
systems.  This  design  application  of  the  APM  was  examined  in  the  context  of  a 
training  syrtem  for  the  new  (to  be  selected)  9mm  Army  handgun.  The  third 
major  activity  during  year  2  was  to  examine  the  potential  for  computer-aiding 
the  A?M  application  procedure.  For  that  effort,  a  sample  application  of  the 
APM  was  programmed  on  an  Apple  II  computer,  in  order  to  demonstrate  an 
ability  to  automate  the  routine  procedures  as  an  interactive  analytic  process, 
thereby  making  the  APM  application  a  cost-effective  and  efficient  technique. 

The  rationale  behind  the  work  on  this  project  rests  with  the  problem  that 
te*  lets,  analyzers  and  researchers  too  often  use  an  incomplete  or  inappropriate 
set  of  human  performance  measures  in  evaluating  or  specifying-  a 
human-machine  (e.g.,  training)  system.  Those  are  usually  known  measures 
often  selected  without  adequate  consideration  of  the  system  context,  which  may 
no!  help  clearly  assess  system  performance  ar-’  may  not  provide  adequate 
answers  to  the  essential  questions  about  system  ffectiveness.  Because  there 
is  no  verified  analytic  process  for  deriving  (>  specifying)  the  optimal 
measures  of  a  system's  performance  or  effectiveness,  true  assessment  needs 
arc  difficult  to  define  and  the  process  is  relatively  haphazard.  The  typical 
so'ution  is  to  test/measure/spccify  the  easy  and  accessible  system  points,  but 
not  necessarily  those  that  should  be  addressed.  Without  having  more 
systematic  procedures,  people  measure  or  specify  what  can  be  counted  (e.g., 
POi  hours),  or  observed  (e.g.,  number  of  troops  trained).  They  design 
written  tests  to  assess  facts  rather  than  understanding,  and  use  criteria  such 
as  end-nf-courso  tests  rather  than  on-the-job  performance.  Rarely  do  people 
know  the  relationship  between  test  performance  and  job  performance,  or 
between  soldier  job  performance  and  unit  effectiveness. 

'‘(Ton Ira cFTItTe*:  "Effectiveness  of  Infantry  Systems:  TEA,  CTEA,  and  Human 
Factors  in  Systems  Development  and  Fielding." 

1  Bloc- 1,  U.F.,  Oat.ec,  J.F.,  Jr.  and  Hamilton,  J.W.  An  Initial  Analytic 
Prom  s'.  Model  for  Ss-stcms  Measurement ;  Extensions  of  the  Systems  Taxonomy 
liijnMrE;  CT:  Dunlap  and  Associates,  Inc.,  30  April  l98l.  (Final  * 

fJumj.ary  Report). 
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POI  hours),  or  observed  (e.g.,  number  of  troops  trained).  They  design 
written  tests  to  assess  facts  rather  than  understanding,  and  use  criteria  such 
as  end-of-course  tests  rather  than  on-the-job  performance.  Rarely  do  people 
know  the  relationship  between  test  performance  and  job  performance,  or 
between  soldier  job  performance  and  unit  effectiveness. 

The  results  of  those  inadequacies  of  the  measurement  determination 
process  are  the  wasting  of  valuable  resources  (time,  effort,  talent,  money), 
the  failure  to  provide  adequate  answers  to  effectiveness  questions,  and  the 
relegation  to  obscurity  of  the  elusive  questions  regarding  human  contributions 
to  system  performance.  A  better  method  is  required  to  decide  what  should  be 
measured  and  how.  Hence,  the  development  of  this  analytic  process  model  for 
system  design  and  measurement. 

The  APM  is  intended  to  enable  testers,  analysts  and  researchers  to  define 
system  factors  or  taxonomies,  and  to  translate  taxa  into  measures  or  design 
requirements.  The  model  forces  one  to  describe  the  system  of  interest  and  its 

human  elements  in  such  a  way  as  to  suggest  a  more  complete  set  of 

system-human  attributes  pertinent  to  system  effectiveness.  The  model  aids  the 
users  by  providing  general  "menus"  of  factors  (taxa),  and  procedures  to  help 
translate  those  taxa  into  appropriate  measures  or  design  requirements.  The 
APM  for  measurement  is  illustrated  in  Figure  1  (the  APM  for  design  has  not 
yet  been  defined  this  completely).  The  five  blocks  (numbers  1  through  5) 
across  the  top  of  Figure  1  comprise  the  most  highly  developed  portion  of  the 
model,  and  are  often  referred  to  as  the  systems  taxonomy  model  (STM). 
During  this  second  year  of  work,  more  development  effort  than  before  was 

applied  to  blocks  6  and  7  in  an  attempt  to  implement  more  completely  what  is 

considered  to  be  the  basic  mission  of  the  model  under  the  present 
contract — routinizing  the  derivation  of  appropriate  measures  and  design 
requirements.  The  remaining  blocks  of  Figure  1  (numbers  8  through  17)  have 
received  little  attention  in  this  study  because  our  present  objectives  of 
developing  procedures  for  deriving  the  required  information  are  met  if  one  can 
complete  the  model  steps  through  block  7.  The  APM  for  design  specification  is 
most  similar  to  the  measurement  version  (Figure  1)  in  the  early  stages 
(taxonomy  development),  and  becomes  less  similar  in  the  later  stages. 

Details  of  the  APM  and  significant  accomplishments  in  its  continuing 
development  and  application  are  described  in  the  following  chapter.  Specific 
applications  of  the  model  to  training  systems,  and  their  measurement  and  initial 
design  are  described  in  the  later  chapters.  Following  that  is  a  description  of 
the  computer-aided  application  of  the  model  and  future  plans. 
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II.  THE  ANALYTIC  PROCESS  MODEL  (A PM) 


A  brief  summary  of  significant  APM  concepts,  definitions  and  development 
milestones  are  provided  here  as  a  departure  point  for  the  reader  in  reviewing 
the  results  of  our  second  year's  work. 

A .  Initial  APM  Concepts  and  Definitions 

The  initial  work  on  manned  sy^t^m  measurement  and  the  systems  taxonomy 
model  by  Finley  and  her  colleagues  '  indicated  that  certain  prerequisites  exLst 
for  including  "system"  factors  in  manned  system  performance  measurement. 
They  are:  (1)  recognition  of  systems  as  viable  entities  in  and  of  themselves; 
and  (2)  development  of  conceptual  tools  for  the  purpose  of  grouping  systems 
into  populations,  defining  these  populations,  and  placing  them  into  a  context 
with  other  populations. 

In  al!  cases,  a  taxonomy  supplies  knowledge  that  is  specifically  relevant 
to  the  analytic  application  at  hand.  Thus,  each  system  taxonomy  is  unique  to 
the  particular  system  and  to  the  particular  context  and  purposes  in  and  for 
which  the  measurement  or  design  process  is  to  be  applied.  What  Finley  et  al. 
sought  was  a  systematic  way  of  generating  such  taxonomies  for  any  given 
system  and  analytic  purpose.  Development  of  the  STM  and  its  encompassing 
APM  by  the  Dunlap  staff  in  the  current  project  was  intended  to  help  meet  that 
need.  A  description  of  work  ^completed  during  the  first  year  of  this  project  is 
found  in  Bloom,  et  al.  (1981  r,  and  is  summarized  below.  The  reader  will  note 
that  the  emphasis  in  this  early  APM  development  work  has  been  on  the  STM 
portion.  Later  stages  of  the  APM  are  addressed  in  the  second  year's  effort. 

First,  system  populations  can  be  formed  according  to  a  context  factor 
consisting  of  the  following:  the  kind  of  system  each  population  member  is 
supposed  to  be;  the  kind  of  mission  or  job  each  member  is  supposed  to  be  able 
to  do;  the  circumstances,  conditions,  and  constraints  under  which  each  member 
is  supposed  to  work ;  the  specific  requirements  and  criteria  each  member  is 
supposed  to  satisfy;  etc.  Each  such  population  immediately  suggests  design 
and  measurement  issues,  such  as:  Is  the  system  what  it  is  supposed  to  be? 
Can  it  do  what  it  should?  Will  it  work  where  it  is  supposed  to? 


Finley,  D.L.,  Muckier,  F.A.,  Gainer,  C.A.  and  Obermayer,  R.W.  An 
Analysis  and  Evaluation  Methodology  for  Command  and  Control:  Final 
Technical  Report^  FTorthridge,  CA:  Manned  Systems  Sciences,  Inc., 
^November  19/5. 

Finley,  D.L.  and  Muckier,  F.A.  Human  Factors  Research  and  the 
Development  of  a  Manned  Systems  Applications  Science:  The  System  Sampling 
Problem  and  a  Solution.  Korthridge,  CA:  Manned  Systems  Sciences,  Ir, c .  , 
-Ttlfy 'T976. - 


Bloom,  et  al.,  1981.  O 


A  second  factor  by  which  any  system  can  be  examined  is  that  of 
descriptive  level,  which  refers  to  the  several  levels  of  increasing  detail  into 
which  a  system  can  be  divided.  First,  a  system  can  be  described  as  an 
indivisible,  macroscopic  entity,  or  "black  box."  This  overall  level  of 
description  can  be  thought  of  as  focusing  on  the  system's  basic  objectives. 
Taxa  that  would  be  identified  on  that  level  of  description  would  be- expecTed  to 
generate  nominal  specifications  or  taxa  of  performance/effectiveness,  and  would 
tend  to  be  particularly  directed  to  generalizable  research  issues.  Second,  the 
system  can  be  described  in  more  detailed  terms  of  how  it  can  be  applied,  i.e., 
its  functional  purposes.  This  level  of  description  probably  would  produce 
taxonomies  directed  to  both  fundamental  and  system-specific  types  of  research 
questions,  and  would  generate  both  nominal  and  relative  measures.  Third,  a 
system  can  be  described  in  terms  of  how  it  can  achieve  its  purposes,  i.e.,  in 
terms  of  the  characteristics  of  its  operations.  This  level  of  description  would 
tend  to  yield  relative  measurements,  and  be  applicable  mainly  to  specific 
research  issues.  The  distinction  between  generic  and  specific  research  issues 
will  be  illustrated  later  in  Section  III  (Figure  13). 

These  first  two  factors  (system  context  and  levels)  suggest  a  descriptive 
matrix  for  the  model,  which  in  its  rudimentary  form  served  as  a  point  of 
departure  for  the  work  under  this  contract.  In  its  subsequent,  evolved  form, 
shown  in  Figure  2,  the  matrix  served  as  a  point  of  departure  for  the  second 
year's  work. 

Because  any  system  tends  to  be  associated  with  larger  and  smaller 
systems  in  carrying  out  its  operations,  that  system  of  interest  must  be  viewed 
in  terms  of  those  operating  relationships.  Consequently,  a  third  factor  of  the 
APM  is  the  hierarchical  structure  of  system  operation,  illustrated  in  Figure  3. 
This  third  factor  is  intended  to  insure  that  proper  attention  to  interacting 
entities  is  given  in  any  system  design  or  measurement  process.  The 
hierarchical  structure  includes  every  system  with  which  the  system  of  interest 
dire  *ly  interacts.  In  general,  a  system  interacts  directly  with: 

Larger  systems,  of  which  it  is  a  part,  and  superior  systems,  to 
which  it  is  subservient  in  a  command/control  sense;  collectively, 
these  larger  and/or  superior  systems  are  termed  suprasystems  in  the 
hierarchical  structure. 

Smaller  systems  that  comprise  it,  and  inferior  systems  over  which  it 
exercises  command/control;  collectively,  these  are  termed 
subsystems . 

Systems  that  exist  and  operate  on  their  own  level  of  command  and 
control,  with  which  they  share  resources  and/or  exchange 
mput/output;  collectively,  these  are  called  collateral  systems. 

When  the  two-factor  matrix  (Figure  2)  is  applied  to  the  system  of  interest 
and  its  associated  hierarchy,  the  model  takes  on  a  three-factor  appearance  as 
shown  in  Figure  4.  Each  cell  of  each  matrix  of  Figure  4  yields  a  list  or 


Figure  4.  The  Three-Factor  Structure  of  the  A  PM  (STM  Segment  Only) 
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taxonomy  of  descriptors  pertaining  to  the  system  being  analyzed.  Each  item  or 
taxon  in  each  list  is  a  potential  basis  for  design  specifications  or  evaluation 
measures.  One  can  readily  see  the  enormous  magnitude  of  effort  that  could  be 
required  in  applying  this  three-factor  model  even  to  a  simple  system.  Hence, 
an  increasingly  important  part  of  this  study  is  the  attempt  to  aid  and 
accelerate  Ihe  model  application  by  using  an  interactive  computer.  The 
computer  can  lead  the  analyst  through  the  model  steps  in  a  systematic  fashion 
by  asking  the  proper  questions,  by  providing  clarification  and,  especially,  by 
offering  the  analyst  general  taxonomies  and  measures  that  can  be  used 
directly,  ignored,  modified  or  supplemented. 

B .  Model  Development,  Prior  Applications  and  New  Objectives 

The  overall  goal  of  this  project  is  to  improve  the  capability  for  measuring 
the  effectiveness  and  performance  of  training  and  other  human-machine 
systems.  However,  the  specific  areas  of  emphasis  within  that  goal  have 
undergone  redefinition  and  clarification  during  the  project  since  1980.  In  the 
first  year,  it  was  implicitly  assumed  that  the  research  would  address 
measurement  of  existing  (i.e.,  at  least  prototype)  systems.  In  this  second 
year,  however,  applications  to  system  concepts  also  were  examined.  That  is, 
results  of  the  first  year  suggested  that  tlie  project's  technological 
developments  may  be  useful  not  only  for  evaluating  emerging  and  operational 
systems,  but  also  for  helping  to  define  and  specify  systems  at  the  conceptual 
level,  as  an  input  to  system  design.  Both  such  applications  are  now  included 
within  the  project's  scope  of  activities. 

Also,  while  the  project  continues  to  seek  technological  developments 
suitable  for  use  with  the  total  population  of  human-machine  systems, 
concentration  is  on  training  systems.  It  is  somewhat  different  than  the  Year  1 
model  development  effort,  in  which  the  APM  was  applied  to  the  combat 
performance  of  an  emerging  weapon  system,  the  Bradley  Infantry  Fighting 
Vehicle.  In  particular,  the  sample  BIFV  function  that  was  most  intensively 
analyzed  with  the  model  for  measurement  implications  was  that  of  Surveillance. 
Applications  to  systems  of  lesser  complexity  were  also  tried  in  Year  I,  in  an 
effort  to  clarify  the  elementary  stages  and  procedures  required  of  the  analytic 
process  being  modeled.  By  the  time  the  project  began  its  second  year,  there 
were  three  major  accomplishments  upon  which  to  build: 

0  A  comprehensive  analysis  of  the  state  of  art  of  human-machine 

system  measurement,  including  annotated  bibliography  of 

literature  pertaining  to  that  field. 

c  An  improved,  extended  Systems  Taxonomy  Model,  developed  and 
verified  through  trial  applications  to  systems  of  increasing 
complexity. 


Edwards,  J.M.,  Bloom,  R.F.,  Oates,  J.F.,  Jr.,  Sipitkowski,  S.,  Brainin, 
P.A.,  Eckenrode,  R.J.  and  Zeidler,  P.C.  An  Annotated  Bibliography  of  the 
Manned  Systems  Measurement  Literature.  Darien,  CT:  Dunlap  and  Associates, 
Inc.,  30  November  1^81. 
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An  outline  of  the  overall  process  of  human-machine  system 
measurement,  providing  the  skeletal  structure  for  a  total  analysis 
process  model  (APM).  The  STM  constitutes  the  first  stage  of  that 
APM. 

This  second  year's  work  was  designed  to  continue  development  of  the 
APM,  proceeding  from  the  first  stage  (STM)  to  at  least  the  second  stage 
(blocks  6  and  7,  Figure  1)  of  the  overall  process.  Previous  accomplishments 
demonstrated  that  a  face-valid  hierarchy  of  measures  could  be  derived  from  the 
taxonomies,  but  explicit  procedures  and  guidelines  for  deriving  those  measures 
were  yet  to  be  developed.  One  objective  of  the  Year  2  work  was  to  begin 
developing  those  procedures  and  guidelines. 

A  second  objective  in  Year  2  was  to  examine  the  applicability  of  the  model 
to  the  definition  and  specification  of  new  systems.  Third,  the  project  was  to 
address  the  issue  of  streamlining  the  process  of  applying  the  STM  (and  APM). 
In  particular,  the  potential  for  implementing  the  first  portion  of  the  APM  as  a 
computer-aided  model  was  to  be  explored  to  a  reasonable  depth. 

Six  tasks  were  specified  and  completed  for  the  Year  2  effort.  First,  the 
STM  itself  was  further  developed,  through  improvement  of  the  taxonomization 
guidelines  and  the  identification  of  comprehensive,  usable  sets  of  taxonomies 
(when  possible)  for  the  model's  cells.  This  further  development  was  extended 
to  the  procedures  for  deriving  measures  hierarchies  once  taxonomies  are 
identified.  Also  addressed  briefly  were  concepts  and  procedures  for: 

(1)  sorting  out  taxonomies  and  measures  on  a  "system  vs.  subsystem"  basis; 

(2)  differentiating  measures  of  "effectiveness"  vs.  "performance"  in  identified 
hierarchies;  and  (3)  specifying  the  human  contribution  to  any  given  system 
measure. 

Second,  the  model  was  applied  to  one  aspect  of  an  existing  system  as  a 
further  test  of  its  utility  and  validity  for  aiding  systems  measurement.  The 
system  of  interest  for  that  trial  application  was  the  BIFV  Training  System. 
The  analytic  "purpose"  selected  was  to  help  measure  that  aspect  of  gunnery 
training  design  concerned  with  the  specification  of  gunner-learner  testing. 

The  third  task  was  to  apply  the  model  to  a  selected  system  concept,  to 
test  its  utility  in  providing  input  to  system  definition  and  specification.  The 
system  selected  for  this  application  was  the  training  system  for  the  new  9mm 
Army  handgun.  The  analytic  "purpose"  selected  was  to  help  specify  that 
aspect  of  the  handgun  training  system  concerned  with  the  design  of  the 
curriculum. 

As  the  fourth  task,  the  results  of  the  two  major  trial  applications  were 
examined  in  detail,  and  the  needs  for  additional  model  development  were 
identified. 

The  fifth  task  was  to  review  the  potential  for  implementing  a 
computer-aided  version  of  the  extended  STM.  This  task  produced  a  working 
(but  reduced-scale)  computer-aided  model,  suitable  for  demonstration 
purposes. 
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The  final  task  was  to  document  the  results  of  the  second  year's  work. 
The  principal  documents  produced  were  a  working  paper  describing  the  results 
of  the  trial  application  to  the  9mm  handgun  training  system  concept,  and  the 
present  technical  report  providing  details  on  all  of  the  project's  tasks.  As  a 
third  item  of  "documentation,"  all  products  of  the  computer-aided 
implementation  were  delivered,  including  hard  copy  listings  and  magnetic  disk 
copies  of  all  programs  and  data  bases  (taxonomies),  and  equipment  user 
manuals. 

This  second  year's  effort  contributed  to  the  longer  term  goals  of 
producing  t 

°  Automated  procedures  to  help  derive  training  systems  effectiveness 
measures. 

°  Demonstrated  applicability  and  usability  of  the  APM  by  Army  users. 

°  Demonstrated  utility  of  the  APM  for  specifying  new  training  systems 

development  needs. 

0  Guidelines  and  procedures  for  user  training  and  user  application  of 
the  automated  APM. 

It  is  envisioned  that  the  users  of  the  APM  for  design  and  measurement 
purposes  will  be  at  the  U.S.  Army  Infantry  School  (USAIS)  and  other  schools 
of  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  including  the 
Directorate  of  Training  Development  (DTD),  the  Directorate  of  Evaluation 
(DOE),  and  the  Directorate  of  Combat  Development  (DCD).  APM  users  are 
also  foreseen  to  include  personnel  with  the  U.S.  Army  Test  Boards,  as  well  as 
the  U.S.  Army  Operational  Test  and  Evaluation  Agency  (USAOTEA)  .and,  of 
course,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI).  Among  the  individuals  at  those  agencies  who  are  seen  as  the 
ultimate  users  of  automated  APM  are  such  types  as  training  developers  and 
evaluators,  TRADOC  system  managers  (TSMs),  system  researchers  and 
designers,  analysts  and  planners. 

The  computer-aided  APM  will  help  realize  the  ultimate  objective  of 
providing  the  above-noted  users  with  a  uniform,  thorough,  adaptive  and 
efficient  procedure  to  aid  in  the  process  of  deriving  the  most  meaningful 
design  specification  requirements,  design  specifications,  evaluation  measurement 
requirements  or  evaluation  measures  for  any  planned  or  existing 
human-machine  system,  but  particularly  one  in  the  area  of  training  systems. 
Progress  toward  that  end,  as  achieved  in  Year  2  of  this  contract  work,  is 
described  next. 


III.  APPLICATIONS  OF  THE  APM  TO  TRAINING  SYSTEMS 


The  narrowed  focus  of  the  project  from  the  total  population  of 
human-machine  systems  to  the  subpopulation  of  training  systems,  permitted 
considerable  progress  to  be  achieved  that  might  not  have  been  possible 
otherwise.  Clear  delineation  of  a  system's  hierarchical  structure  has  been 
shown  to  be  a  sine  qua  non  for  the  identification  of  performance  taxa  and  the 
application  of  the  APM'.  As  long  as  the  focus  remained  on  systems-in-general, 
nothing  specific  could  be  said  about  system  structure.  The  "general  system's" 
hierarchy  (Figure  3,  earlier)  is  a  non-specific  conceptualization  of 
suprasystems,  subsystems,  and  collateral  systems,  interlinked  in  some 
undefined  fashion.  However,  the  shift  of  interest  to  one  particular 
sub  population  immediately  allowed  a  tractable  structure  to  be  defined  in 
considerable  detail.  In  other  words  and  without  extreme  oversimplification,  it 
can  be  said  that  all  training  systems  share  a  common  hierarchical  structure, 
consisting  of  identical  subsystems  and  analogous  collateral  systems  and 
suprasystems;  that  these  structural  members  can  be  defined  in  detail  for 
purposes  of  APM  application;  and  that  general  training  system  performance 
taxa  can  be  identified  and  subsequently  applied,  for  measurement  or  design 
specification,  to  any  training  system  of  interest.  Identification  of  those 
general  performance  taxa  occupied  a  significant  segment  of  the  research  effort 
expended  during  the  project's  second  year.  Important  tools  now  are  becoming 
available  for  evaluation  and  design  application  involving  this  key  class  of 
human-machine  systems.  Those  tools  include  a  set  of  general  performance 
attributes  for  all  training  systems,  with  computer-aided  procedures  for 
translating  them  into  effectiveness  measures  or  design  requirements  in  specific 
training  systems. 

This  second  year  of  research  probably  would  have  followed  a  similar  path 
and  led  to  similar  products  if  the  focus  had  narrowed  to  any  other 
subpopulation  of  systems.  However,  it  seems  particularly  appropriate  to  have 
chosen  training  systems  as  the  field  of  interest,  for  at  least  these  reasons; 

°  Training  systems  are  pervasive.  Virtually  every  other  kind  of 

human-machine  system  requires  a  corresponding  training  system  to 
insure  its  operability.  Training  very  likely  forms  the  largest  single 
subpopulation  of  systems,  and  thus  represents  the  largest  potential 
base  of  applications  for  the  APM. 

0  Training  systems  are  human-focused,  probably  to  a  greater  degree 
than  is  any  other  system  subpopulation.  Training  system 
performance  has  an  especially  large  component  of  human 
performance.  Given  ARI's  fundamental  focus  on  the  behavioral  and 
social  sciences,  training  is  an  appropriate  sphere  of  interest  for  this 
ARI  research  effort. 


o 


Training  is  one  area  of  human  endeavor  for  which  rapidly  expanding 
technology  is  available,  but  only  slowly  or  seldom  is  that  technology 


*  - 

applied.  Training  system  designers  often  seem  to  be  bound  by 
tradition  and  familiar  practice  to  a  greater  degree  than  are  their 
counterparts  in  other  (e.g.,  data  processing)  systems.  Newer, 
potentially  more  effective  and  less  costly  methods  of  individualized, 
self-paced,  on-site  training  often  seem  to  be  overlooked  in  favor  of 
the  tirne-honored  collective,  group-paced,  instructor-dominated 
academies.  An  analytically  rigorous  and  performance-oriented  model 
such  as  the  APM  should  facilitate  the  modernization  of  training 
system  design  and  help  direct  training  evaluators  to  issues  of  real, 
current  significance. 

0  Important  new  Army  training  systems  are  presently  under 

development.  Ranging  from  training  to  support  use  of  individual 
weapons,  through  those  associated  ^with  fighting  vehicle  operations  to 
those  linked  with  high  level  C  (command,  control  and 
communications)  operations,  these  new  training  systems  will  have  a 
major  impact  on  the  Army's  overall  combat  effectiveness.  By 
insuring  the  APM's  applicability  to  training,  this  present  research 
project  may  provide  important  assistance  toward  that  larger  issue. 

Accepting,  then,  that  training  is  a  valid  and  desirable  context  in  which 
to  continue  APM  development  and  application,  the  project  staff  undertook  the 
tasks  of: 

0  Constructing  a  human-machine  systems-oriented  view  of  training, 
i.e.,  a  view  compatible  with  the  APM's  fundamental  structural 
concepts; 

;  identifying  taxa  of  performance  for  key  elements  of  a  generalized 
training  system; 

0  Applying  those  generalized  taxa  to  emerging  Army  training  systems, 
to  assess  their  validity  and  their  utility  for  system  measurement  and 
design  specification; 

Investigating  the  feasibility  of  streamlining  and  expanding  the  APM's 
applicability  to  other  training  systems,  through  automation  of  certain 
procedures . 

The  work  and  results  are  described  below. 

A .  Training  Systems  in  General:  Axioms  for  APM  Application 

The  APM  was  developed  for  application  to  any  human-machine  system.  In 
order  to  apply  the  model  to  the  training  sphere,  it  must  be  possible  to  show 
that  a  training  system  (as  a  general  concept)  possesses  the  essential 
characteristics  of  a  human-machine  system,  and  that  the  structure  of  a 
training  system  conforms  to  the  general  architecture  of  all  systems.  It  is 
appropriate  and  instructive  to  begin  by  examining  training  in  light  of  a 
general  human-machine  systems  concept,  in  order  to  explore  how  that  concept 
is  reified  for  this  particular  subpopulation. 


-13- 


One  immediately  encounters  a  stumbling  block:  "human-machine  system" 
is  one  of  those  technical/philosophical  notions  that,  although  critically 
important  for  research  and  scholarship,  are  seldom  defined.  The  mathematical 
concept  of  a  "set"  is  another  of  these.  When  attempts  have  been  made  to 
define  the  set  concept  rigorously,  enlightenment  and  progress  sometimes  have 
resulted,  but  usually  at  the  price  of  paradox,  limitations  in  the  utility  of  the 
concept,  and  disagreement  among  scholars  holding  to  divergent  rigorous 
definitions  of  "set."  Most  mathematicians  accordingly  prefer  to  deal  with 
so-called  naive  set  theory,  in  which  no  attempt  is  made  to  define  the 
fundamental  concept  precisely.  Instead,  they  adopt  the  implicit  attitude  that 
they  "know  a  set  when  they  see  one,"  and  cheerfully  go  about  the  business  of 
achieving  great  breakthroughs  in  their  research. 

A  human-machine  system  is  something  else  that  people  usually  feel  they 
know  when  they  see  one.  The  absence  of  precise  definitions  fosters  a  loose 
and  often  confusing  use  of  terms.  Consider,  for  example,  the  term 
"subsystem."  Purists  (including  the  authors  of  this  report)  insist  that, 
whatever  else  a  subsystem  may  be,  it  is  itself  a  system.  That  is,  it  must 
possess  exactly  the  same  essential  characteristics  that  permit  any  system  to  be 
so  termed.  However,  one  often  hears  references  to  the  "personnel 
subsystem,"  which  has  come  to  connote  simply  the  people  who  man  a  system  of 
interest,  but  not  its  equipment  or  other  components.  Further,  most 
researchers  will  agree,  at  least  in  the  abstract,  that  a  "subsystem"  must  be 
totally  contained  within  the  (larger)  system  to  which  it  is  subservient.  Yet  in 
practice  one  often  encounters  entities  termed  subsystems- tKat  are  at  least 
partly  outside  the  system  of  interest.  For  example,  it  is  common  to  hear 
speakers  refer  to  the  "maintenance  subsystem"  of  an  armored  vehicle  when 
they  intend  to  include  the  personnel,  equipment,  and  facilities  at  the  direct 
support  and  general  support  levels  of  maintenance:  components  and  people 
clearly  outside  the  scope  of  the  operating  vehicle  system  itself. 

It  is  not  the  intent  of  the  authors  to  impose  rigorous  definitions  upon 
systems  theory.  The  purposes  of  thi^  research  are  quite  compatible  with  a 
naive  human-machine  systems  science  in  which  the  entities  of  interest  are 
known-when-seen .  However,  it  is  crucial  that  the  readers  of  this  report  at 
least  have  some  idea  of  what  the  authors  see  when  they  look  at  a 
human-machine  system.  As  a  fundamental  axiom,  the  research  documented 
herein  presupposes  that  a  human-machine  system  can  be  described  (not 
defined)  as: 

A  collection  of  people,  equipment,  and  procedures  that 
work  together  to  accomplish  specified  functions. 

No  attempt  is  made  here  to  impose  precision  on  the  notions  of  people, 
equipment,  procedures,  collections,  functions,  or  on  the  concept  of  "working 
together."  All  of  these  things  are  treated  naively  as  things  one  "knows  when 
they  are  seen."  Nevertheless,  some  helpful  insights  can  be  gleaned  from  this 
naive  axiom: 

1.  There  must  be  at  least  one  human  in  a  human-machine  system. 

Further,  the  humans  have  to  take  an  active  part  in  the  work:  they 
can't  simply  be  "passengers." 
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There  must  be  machines  in  a  human-machine  system.  They  too  must 
have  an  active,  useful  role  in  the  system's  working:  they  can't 
simply  be  "freight." 


gy 


3.  There  must  be  a  purpose  to  a  human-machine  system.  The  humans 
and  machines  can't  simply  interact  blindly,  unconcerned  about  any 
chance  results  that  might  occur.  Their  functions  must  be  specified 
at  the  outset.  It  may  be  remotely  within  the  realm  of  possibility 
that,  given  enough  time,  ribbon  and  paper,  a  baby  playing  with  a 
typewriter  could  reproduce  the  Aeneid.  That  would  be  remarkable, 
but  there  wouldn't  be  a  human-machine  system  in  operation  unless 
precisely  that  result  had  been  intended. 

If  the  reader  will  accept  this  loose,  axiomatic  view  of  the  world  of 
human-machine  systems,  it  becomes  possible  to  ask:  Does  training  fit  this 
system  view?  What  does  one  see  when  one  looks  at  a  training  system?  V/ hat 
are  its  purposes,  or  specified  functions?  Who  are  its  actively  working  people? 
What  is  its  functioning  equipment?  What  are  its  procedures?  How  are  the 
people,  equipment  and  procedures  organized,  and  how  do  they  work  together? 

The  authors  assert  as  another  axiom  that  any  training  system  works  to 
accomplish  two  fundamental  purposes,  or  master  functions:  learning  and 
helping-to-learn.  These,  too,  are  terms  often  defined  differently  by  different 
users.  In  this  report,  the  classic,  broad  definitions  are  employed: 

°  Learning  is  any  activity,  involving  the  senses,  that  affects  behavior. 

Thus,  in  any  training  system  there  must  be  activities  requiring  use 
of  a  person's  sight,  hearing,  smell,  taste,  feeling  or  any  combination 
of  senses,  and  those  activities  ultimately  must  impact  in :  some 
intended  fashion  on  what  the  person  does  or  how  the  person  does 
something . 

c  Helping-to-learn  is  simply  the  process  of  providing  an  efficient 
learning  environment. 

In  any  training  system,  steps  must  be  taken  to  make  it  as  conducive 
as  possible  for  the  learner  to  learn. 

Very  simply,  then,  a  training  system  exists  if  and  only  if  somebody 
learns  and  somebody  or  something  helps  that  person  to  learn.  It  might  be 
that  the  person  doing  the  learning  and  the  person  doing  the  helping-to-learn 
are  the  same  individual  and  no  one  else  is  involved.  It  might  be  that  many 
people  learn  simultaneously  and  in  concert  with  one  another,  under  the 
guidance  of  another  individual,  or  with  automated  machinery  providing  the 
helping.  It  might  be  that  many  individuals  interact  dynamically  in  an 
atmosphere  of  mutual  inquiry,  each  one  learning  and  helping  the  others  to 
learn,  with  no  one  exclusively  playing  the  role  of  learning  director.  In  any 
event,  the  people,  equipment  and  procedures  must  work  together  to  see  to  it 
that  the  intended  learning  and  helping-to-learn  take  place. 
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N  <  x t ,  who  are  the  people  involved  in  a  training  system?  In  m  re  familiar 
terms,  who  opr- rates  a  training  system? 

It  is  self-evident  that  any  Gaining  system,  no  matter  how  prosaic,  must 
have  at  least  one  type  of  operator:  the  person  or  persons  who  do  the 
learning.  Training  systems  always  strive  to  insure  that  human  beings  learn 
something.  Helping  equipment  or  machinery  to  "learn"  is  the  province  ot 
cybernetics  and  programming  systems.  The  time-honored  terms  for  the  hurrnn 
who  learns  in  a  training  system  are  "student"  or  "trainee".  However,  the-;** 
terms  have  connotations  that  subtly  impede  the  marriage  of  training  systems 
with  modern  technology.  "Student"  connotes  a  subordinate  in  the  traditional 
classroom  setting,  for  whom  the  principal  sensory  activity  is  listening  while 
keeping  hands  still  and  mouth  shut.  "Trainee"  also  connotes  a  receptive, 
subservient  role  (where  there  is  a  "trainee,"  one  expects  to  find  a  "trainer," 
and  there  is  little  doubt  as  to  who  is  in  charge).  Further,  the  term  "trainee” 
conjures  up  the  image  of  a  complete  novice,  a  virtual  tabula  rasa.  In  reality, 
learning  activities  often  are  essential  for  people  who  already  know  a  good  deal 
about  the  subject  matter  of  interest  In  an  effort  to  remove  the  excess 
semantic  baggage,  the  term  "learne  or  "learners"  will  be  used  throughout 
this  report  to  designate  the  essential  training  system  operator  or  unified  group 
of  operators. 

As  suggested  previously,  it  is  possible  for  a  valid  training  system  to  be 
operated  entirely  by  a  single  individual.  In  that  case,  the  learner  also  is  the 
only  human  being  responsible  for  helping  himself  or  herself  to  learn.  (Legend 
has  it  that  Dizzy  Dean  taught  himself  to  pitch,  totally  unaided.  His  practice 
consisted  of  hurling  a  homemade  facsimile  of  a  baseball  at  a  peach  basket 
hanging  from  a  tree  limb.)  In  many  cases,  however,  other  human  operators 
are  involved,  even  if  only  behind  the  scene.  A  group  of  learners  frequently 
are  involved  together  in  training  as  a  collective  group,  unit  or  team  (e.g., 
Individual  Collective  Training  Plan). 

Who  are  these  other  operators?  Since  the  only  two  functions  of  a  training 
system  are  learning  and  helping-to-learn ,  any  operators  who  aren't  learners 
must  be  learning  helpers,  i.e.,  those  who  work  to  provide  an  efficient  learning 
environment.  Several  subclasses  of  learning  helpers  might  participate  in  any 
given  training  system,  including: 

0  Training  administrators.  These  are  the  people  responsible  for 

determining  the  basic  needs  for  learning  and  for  exercising  command 
and  control  over  the  training  system  designed  to  satisfy  those 
needs . 

0  Curriculum  developers.  These  are  the  people  who  plan  the  sensory 
learning  activities  that  will  fulfill  the  identified  needs,  determine  the 
resources  of  material,  equipment,  facilities  and  people  needed  to 
implement  the  activities,  and  assemble  the  plans  and  resource 
specifications  into  a  workable  package. 

°  Facilities  developers.  These  are  the  people  responsible  for 

acquiring,  fabricating ,  and/or  preparing  the  hardware,  facilities, 
equipment,  etc.,  needed  to  implement  the  curriculum. 


°  Logistics  supporters.  These  are  the  people  who  handle  the  mundane 
yet  absolutely  essential  tasks  incidental  to  training  system  operation, 
e.g.,  maintenance  of  equipment,  cleaning  of  facilities,  preparation 
and  provision  of  meals,  transportation,  record-keeping,  etc. 

°  Learning  helpers  who  interact  directly  with  the  learners.  About 
whom,  more  will  be  said  shortly. 

From  the  viewpoint  of  the  learner,  the  first  four  classes  of  learning 
helpers  listed  above  should  be  unobtrusive  in  the  extreme.  Those  people 
should  be  all  but  invisible  to  the  learner,  their  interactions  with  the  learner 
kept  subtle  and  subliminal.  At  its  best,  their  work  shouldn't  really  be  noticed 
by  the  learner.  Like  a  baseball  umpire,  they  will  be  noticed  only  when  there 
is  some  fault  to  find  with  their  performance.  To  cite  a  simple  example, 
suppose  training  is  being  conducted  over  a  period  of  several  days  in  a  meeting 
room  of  a  hotel  isolated  from  other  facilities.  The  hotel's  restaurant,  with  its 
cooks,  waiters,  waitresses,  menu,  oven,  wine  cellar,  etc.,  constitutes  an 
element  of  the  training  system.  Those  cooks  and  other  restaurant  personnel 
aren't  likely  to  think  of  themselves  as  trainers  in  even  the  broadest  sense,  but 
their  work  could  have  a  significant  impact  on  the  comfort,  morale,  and  even 
the  physical  well-being  of  learners  and  other  training  system  operators.  In 
short,  they  have  something  meaningful  to  do  with  the  efficiency  of  the  learning 
environment . 

Similarly,  as  long  as  the  learning  needs  have  been  identified  accurately 
and  all  of  the  people  and  equipment  needed  to  support  the  learning  activities 
are  present,  the  learner  will  not  notice  the  work  of  the  administrators  or  the 
facilities  developers.  As  long  as  the  specific  learning  objectives  are  relevant 
to  ibe  learning  needs,  and  as  long  as  the  learning  activities  are  well  planned 
and  inherently  interesting  and  illuminating,  the  work  of  the  curriculum 
developers  will  remain  unobtrusive.  It  is  the  fate,  even  the  aspiration,  of 
these  kinds  of  training  system  operators  to  be  taken  for  granted.  Their  job  is 
to  do  everything  possible  to  prepare  an  efficient  learning  environment,  and 
then  to  get  out  of  the  way  and  allow  the  learner  to  interact  with  that 
environment. 


The  last  class  of  learning  helpers  is  a  case  apart.  Their  role  is  one 
requiring  direct  contact  with  the  learner.  The  common  terms  for  those  who 
play  that  role  are  very  familiar  and  steeped  in  tradition:  teacher,  instructor, 
professor,  trainer,  etc.  Here  again,  the  connotations  are  not  favorable  to 
modern  training  technology.  Each  of  those  terms  conveys  the  image  of  a 
classroom  dominated  by  an  authoritarian  figure,  actively  imposing  knowledge 
upon  a  group  of  passive  recipients.  The  major  training  functions  postulated 
above  clearly  demand  that  the  learner  be  the  principal  activist  in  the  system: 
it  is  he  or  she  who  must  actually  engage  in  the  sensory  activities  and  it  is  his 
or  her  behavior  that  ultimately  is  to  be  affected.  Furthermore,  much  of 
present-day  training  is  (or  should  be)  tailored  in  pace,  content,  and 
objectives.  Large  classes  marching  in  lock-step  through  identical  learning 
activities  toward  identical  goals  no  longer  represent  the  optimum  training 
modality  for  many  applications.  Thus,  a  better  term  is  needed  to  label  the 


"interacting  learning  helpers,"  a  term  that  connotes  the  supportive  role  these 
key  operators  must  play  and  also  that  connotes  their  essential  responsiveness 
to  the  needs  of  individual  team  learners.  With  some  reluctance,  the  word 
facilitator  is  used  in  this  report.  It  is  a  distastefully  trendy  locution,  but  it 
does  at  least  emphasize  the  learning-helping  nature  of  the  operator's  work  and 
avoids  the  learning-dominating  taint  of  the  older  expressions. 

■To  recapitulate,  the  general  training  system  model  admits  up  to  six  "crew 
classifications,"  viz.,  the  learner  (the  most  important  member  of  the  crew); 
the  facilitator  (a  key  learning-helper  who  interacts  directly  with  the  learner); 
and  the  administrator,  the  curriculum  developer,  the  facilities  developer,  and 
the  logistics  supporter  (all  of  whom  also  are  key  learning-helpers,  but  who 
work  generally  behind  the  learner's  scenes).  It  is  another  fundamental  axiom 
of  this  research  that  the  work  done  by  each  of  these  six  types  of  system 
operators  is  required  in  every  training  system.  In  very  large  training 
systems,  many  individuals  might  be  "assigned"  exclusively  to  each  of  these 
"operator  stations."  In  the  smallest  of  all  training  systems,  one  individual 
might  man  all  of  the  stations  in  a  time-shared  fashion.  Regardless:  if 
learning  is  to  occur,  all  of  their  jobs  must  be  done. 

What  are  those  jobs?  Fundamentally,  each  operator  works  with  other 
operators  and  with  various  system  equipment,  following  prescribed  procedures, 
to  contribute  to  learning  and  helping-to-learn.  Each  does  his  or  her  work  as 
the  key  operator  of  a  particular  training  subsystem,  and  may  play  important 
supportive  roles  in  other  subsystems.  The  description  and  examination  of 
those  subsystems  and  their  performance  requirements  occupy  major  portions  of 
the  remainder  of  this  report.  For  the  present,  the  kinds  of  tasks  required  of 
these  operators  can  be  illustrated  generally  as  follows: 

The  Administrator 

°  Identifies  needs  for  learning 

0  Establishes  an  overall  training  schedule 

0  Allocates  resources  within  the  system 

°  Recruits  and  selects  other  system  operators  (i.e.,  learners, 

facilitators,  curriculum  developers,  etc.) 

°  Monitors  progress 

°  Maintains  records 

°  Evaluates  operator/ subsystem  performance 

°  Evaluates  system  effectiveness 


The  Curriculum  Developer 

°  Establishes  specific  learning  objectives 

°  Determines  training  content 

°  Selects  training  methodologies 

0  Prepares/acquires  plans,  resources,  and  resource 

specifications 

°  Determines  facilitator  requirements 

0  Determines  facilities  requirements 

°  Sets  learner  performance  standards 

0  Establishes  specific  training  schedules 


Th e  Facilities  D ev eloper 


°  Develops/arquires  training  aids  and  devices  required  in  the 
curriculum 

°  Develops/acquires  other  supporting  equipment 

°  Develops/acquires  any  training  sites  and  ranges  required  in  the 
curriculum 

5  Adapts  equipment  and  sites  to  specific  learning  activities 

T  he  Logistics  Supp  or  ter 

°  Provides  for  ammunition,  POL  supplies,  etc. 

°  Provides  for  transportation,  feeding,  lodging,  recreation, 
etc.,  of  all  system  personnel 

Provides  for  housekeeping,  environmental  control  and 
maintenance  of  system  equipment  and  facilities 

The  f  acilitator 

°  Helps  to  motivate  learners  to  learn 

‘  Provides  stimulation  to  facilitate  and  enhance  learning 

°  Provides  day-to-day  management  of  learning  activities 

0  Helps  to  tailor  learning  activities  to  individual  and 

team  needs 

0  Provides  guidance  to  learners  when  needed 

0  Counsels  learners 

c  Tests  and  critiques  iearners1  performance 

The  I  earner 

°  Carries  out  appropriate  sensory  activities  to  achieve 
intended  behavioral  objectives 

The  learner's  job  is  most  succinctly  stated  here,  and  yet  encapsulates  the 
true  purpose  of  the  whole  training  system.  The  other  system  operators  play 
important  roles  requiring  thorough  preparation  and  much  hard,  dedicated 
work.  Eut  their  work  always  is  secondary  to  the  learner's:  a  training  system 
fails  unless  the  learner  actually  learns. 

Next,  what  is  the  equipment  used  in  a  training  system?  What  machinery 
do  the  training  operators  operate? 

Training  system  equipment  is  anything  and  everything  (apart  from  the 
people)  required  for  or  contributing  to  the  learning  activities  and  the  learning 
environment.  This  includes,  most  obviously,  the  equipment,  machinery,  tools, 
etc.,  that  are  used  in  performing  the  job-being-learned.  One  component  of  a 
handgun  training  system,  for  example,  will  be  the  handgun  itself.  Training 
system  equipment  also  includes  learning  resource  material,  e.g,  books, 
pictures,  models,  mock-ups,  films,  tape  recording,  etc.,  used  by  the 
learner  in  the  conduct  of  the  learning  activities.  Equipment  needed  to  support 


or  apply  the  resource  material  would  also  be  included,  e.g.,  chalkboards, 
desks,  tables,  screens,  projectors,  chart  stands,  and  so  forth.  Materials 
needed  to  prepare  and  maintain  the  facilities  also  would  be  part  of  the  training 
system's  equipment  inventory,  as  would  the  facilities  themselves,  whether  they 
be  ranges,  ditches,  classrooms,  bunkers,  study  halls  or  whatever.  All  of  the 
equipment  required  to  provide  logistic  support  also  would  be  included,  e.g, 
kitchen  supplies,  sleeping  accommodations,  vehicles  for  transport,  etc. 

Finally,  what  can  be  said  in  general  about  the  procedures  used  in  a 
training  system0  Procedures  are  the  specific  ways  in  which  system  operators 
work  with  each  other  and  with  the  various  equipment  in  order  to  carry  out  the 
system's  functions.  Given  the  focus  of  a  training  system  on  learning  and 
learning-helping,  a  training  system's  procedures  must  encompass  (among  many 
other  things)  : 

°  How  basic  needs  for  learning  are  identified 
0  How  learning  objectives  are  derived  from  identified  needs 
0  How  learning  activities  are  planned 
0  How  facility  requirements  are  determined 
0  How  facilities  are  acquired  or  developed 

°  How  the  learners  acquire  information 

°  How  the  learners  practice  and  assimilate  learning 

0  How  the  environment  is  controlled 

°  How  the  learners  are  tested 

0  etc. 

Training  systems  typically  document  some  of  their  procedures,  for 
example  in  lesson  plans,  schedules,  etc.  However,  simply  documenting 
procedures  doesn't  guarantee  that  they  will  be  carried  out  as  written. 
Learners,  facilitators,  and  other  operators  often  revise  plans  to  suit -their  own 
tastes.  In  evaluating  training  system  performance,  it  is  important  to  assess 
the  procedures  actually  used,  in  contrast  to  those  officially  specified. 

Training,  then,  is  an  endeavor  that  easily  lends  itself  to  the  concept  of 
human-machine  system.  It  always  involves  people,  equipment  and  procedures 
working  together  to  accomplish  purposeful  functions.  Those  functions  revolve 
around  learning,  i.e.,  the  creation  of  desired  human  behavioral  effects 
through  planned  human  sensory  activities.  All  of  the  performance  capabilities, 
processes,  and  products  built  into  a  training  system  must  enhance  or  support 
learning.  Any  that  do  not  are  at  best  superfluous,  and  possibly  deleterious. 

Given  that  training  systems  are  indeed  human-machine  systems,  what  can 
be  said  about  their  subsystem  structure?  Here  again,  a  "subsystem”  is 
conceived  to  be  a  valid  system,  totally  contained  within  and  subservient  to 
some  larger  system.  The  subsystem's  humans  and  machines  must  all  be 
members  of  the  larger  system,  and  the  subsystem's  purposes  must  be  strictly 
supportive  of  the  larger  system's  purposes. 
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As  a  final  fundamental  axiom,  the  authors  assert  that  every  training 
system  contains  six  subsystems.  Each  subsystem  constitutes  the  "principal 
territory"  of  one  of  the  six  training  operators.  The  operators  also  play 
supportive  roles  in  one  another's  subsystems  as  depicted  in  Figure  5.  The 
names  given  to  the  six  subsystems  intentionally  are  non-traditional.  Just  as 
the  terms  learner  and  facilitator  were  adopted  to  avoid  the  undesirable 
connotations  of  "student"  and  "teacher,"  new  words  are  needed  to  describe  the 
building  blocks  of  training  in  ways  that  do  not  encourage  shopworn  approaches 
to  their  design.  Thus,  for  example,  the  term  "Curriculum  Development"  might 
have  been  used  to  name  the  subsystem  we  have  called  "Design";  however,  the 
older  phrase  carries  with  it  the  stereotypic  classroom,  academy,  professorial 
image.  Similarly,  what  we  have  termed  "Enabling"  is  more  commonly  known  as 
"Instructor  Training."  However,  what  unfortunately  passes  for  instructor 
training  all  too  often  falls  far  short  of  the  facilitator  preparatory  work  that  we 
perceive  as  essential  for  satisfactory  system  performance. 

The  Command  subsystem  is  the  main  territory  of  the  training 
administrator.  The  administrator,  with  help  from  others,  operates  the 
Command  subsystem  to  exercise  control  over  the  total  training  system,  e.g., 
by  recruiting  and  selecting  personnel  to  "staff"  the  other  subsystems,  by 
allocating  resources  to  them,  by  evaluating  their  performance,  etc.  It  is 
within  the  Command  subsystem  that  the  most  fundamental  decisions  concerning 
training  are  made:  decisions  regarding  what  needs  to  be  learned  and  who 
needs  to  learn  it.  Each  of  the  other  subsystems  must  submit  its  plans  and 
products  to  the  Command  subsystem  for  review  and  approval.  Command  thus 
stands  as  a  suprasystem  for  the  other  subsystems  because  of  their 
subservience  to  its  authority. 

The  principal  operator  of  the  Design  subsystem  is  the  curriculum 
developer.  The  developer,  with  help  from  others,  operates  the  Design 
subsystem  to  translate  the  basic  learning  needs  (identified  by  Command)  into 
specific  training  plans.  Design  determines  exactly  what  human  behaviors  are 
to  be  affected  during  learning,  and  specifies  the  sensory  activities  to  be 
conducted  to  achieve  the  effects.  In  discharging  that  responsibility,  Design 
makes  basic  decisions  concerning  the  kinds,  numbers,  and  qualities  of 
facilitators,  equipment,  materials,  and  facilities  needed  to  implement  the 
learning  activities.  Design  produces  all  of  the  procedures,  some  of  the 
materials,  and  all  of  the  resource  specifications  required  for  the  Delivery 
subsystem  (the  subsystem  in  which  learning  actually  takes  place).  Since  it 
thus  provides  essential  input  to  each  of  the  four  remaining  subsystems,  Design 
stands  as  a  collateral  system  for  each  of  them. 

The  Delivery  subsystem  is  the  heart  of  the  structure.  Its  principal 
operator  is  the  learner,  who  operates  Delivery  to  implement  the  prescribed 
sensory  activities  and  thereby  acquire  the  intended  behavioral  effects.  In 
doing  so,  the  learner  is  assisted  by  all  of  the  other  operators,  each  of  whom 
contributes  in  some  way  to  the  efficiency  of  the  learning  activities.  Because  it 
derives  essential  input  from  Design,  Emplacement,  Logistics,  and  Enabling,  thr 
Delivery  subsystem  is  collateral  to  each  of  those  four. 
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Figure  5.  Training  Subsystems  and  Their  Operators 
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The  principal  operator  of  the  Enabling  subsystem  is  the  facilitator.  This 
subsystem  is  unique  in  that  it  is  itself  a  total  training  system:  every  Enabling 
subsystem  plans  and  implements  activities,  involving  the  senses,  that  affect 
behavior  in  a  purposeful  fashion.  The  person  engaging  in  those  activities  is 
the  facilitator,  who  does  so  to  acquire  learning  that  will  prepare  him  or  her  to 
work  in  the  larger  training  system.  The  kinds  of  learning  activities  carried 
out  in  the  Enabling  subsystem  are  intended  to  insure  that  the  facilitator  has 
the  basic  subject  matter  expertise  necessary  to  permit  playing  a  credible  and 
useful  role  in  helping  others  to  learn;  has  the  ability  to  implement  the  kinds  of 
interactions  with  learners  called  for  in  the  training  plans  produced  by  the 
Design  subsystem;  and  has  the  ability  and  the  opportunity  to  tailor  those 
plans  to  meet  the  specific  learning  needs  of  given  learners.  In  short,  the 
Enabling  subsystem  works  to  insure  that  the  training  plans,  procedures,  and 
resources  produced  by  Design  are  made  workable  and  viable  for  each  Delivery 
application.  Among  many  other  things,  activities  might  be  conducted  in  the 
Enabling  subsystem  to  insure  that  facilitators  possess  traditional  platform 
instructional  skills,  but  those  activities  would  be  included  only  if  such 
instructional  skills  are  relevant  to  the  learning  activities  of  the  larger  training 
system.  Even  then,  instructional  skill  training  never  accounts  for  all  of  the 
work  required  of  the  Enabling  subsystem. 

The  Emplacement  subsystem  is  the  main  province  of  the  facilities 
developer.  This  subsystem  discharges  the  responsibility  for  constructing, 
acquiring,  adapting  or  otherwise  preparing  the  equipment  and  materials  needed 
for  the  prescribed  learning  activities,  and  also  for  preparing  the  physical  sites 
at  which  the  activities  are  to  take  place. 

The  Logistics  subsystem's  principal  operator  is  the  logistics  supporter. 
This  subsystem  carries  out  the  work  necessary  to  maintain  the  total  system’s 
humans  and  machines  in  good  working  order. 

Having  postulated  that  all  training  systems  possess  these  six  subsystems, 
it  should  be  possible  to  point  them  out  in  the  operation  of  any  training 
system,  large  or  small.  This  is  done  in  the  following  two  examples. 


EXAMPLE  1.  A  Small  Training  System 

Consider  first  a  bicycle  rider  training  system  operated  by  two  people,  a 
child  who  wishes  to  become  a  rider  and  a  parent  who  agrees  to  help  the  child 
learn.  The  basic  Command  decision,  that  a  need  to  learn  to  ride  exists,  might 
be  initiated  by  the  child  and,  after  some  thought,  approved  by  the  parent. 
The  child's  perceived  need  may  be  strong,  motivated  by  envy  of  already-riding 
playmates,  sharp  attraction  to  a  shiny,  new  bike,  the  desire  for  mobility 
beyond  what  the  feet  or  tricycle  can  provide,  or  whatever.  The  parent  may 
also  perceive  a  need  for  the  child  to  ride,  based  on  the  parent's  desire  to 
express  affection,  provide  happiness,  and  aid  in  the  overall  growing  process. 
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and  also  perhaps  based  on  the  parent's  fond  memory  of  having  learned  to  ride 
himself  or  herself.  However,  the  parent  probably  also  assesses  that  need  in 
light  of  other  needs,  such  as  safety,  economics,  etc.  In  any  event,  the 
decision  is  jointly  made:  if  either  parent  or  child  refuses  to  concede  the  need 
for  the  child  to  ride,  this  bicycle  rider  training  system  will  die  aborning. 
Both  parent  and  child  thus  are  key  operators  of  the  Command  subsystem,  with 
the  parent  (one  hopes)  serving  as  the  higher  ranking  operator. 

If  the  training  is  to  be  successful,  the  parent  had  better  be  the  sole 
operator  of  the  Design  subsystem.  The  child  simply  has  no  way  of  knowing,  a 
priori,  what  behaviors  are  required  for  bike  riding  or  what  learning  activities 
will  lead  to  the  mastery  of  those  behaviors.  The  parent  probably  has  a  fair 
idea  of  what  the  behavioral  objectives  and  learning  activities  ought  to  be, 
based  on  recollection  of  the  parent's  own  bike  training  and  on  a  few  moments' 
reflection  on  the  abilities  needed  for  riding.  Although  the  parent  rarely  will 
express  these  thoughts  in  classic  behavioral  terms,  the  concepts  of  balancing, 
pedaling,  braking,  steering,  turning,  signalling,  etc.,  probably  form  a  loose 
mental  image  of  a  training  content  outline.  The  painstaking  or  extremely 
motivated  parent  might  consult  a  few  bicycle  safety  and 
how-to-teach-your-kid-to-ride  manuals.  The  outcome  of  all  of  this  will  be  a 
very  nebulous  (usually)  and  very  flexible  plan  for  exactly  what  will  take  place 
when  the  learning  actually  starts.  At  roughly  this  same  time  the  parent 
unconsciously  should  "change  hats"  and  start  to  function  as  the  operator  of 
the  Enabling  system.  This  basically  entails  forming  some  rudimentary  ideas 
about  what  will  be  said  to  the  child  (by  way  of  explaining  bicycle  controls  and 
operations),  what  will  be  demonstrated  to  the  child  (to  clarify  what  is 
required),  how  the  child  will  practice  (to  assimilate  learning),  and — very 
roughly-- how  the  parent  will  know  when  the  child  has  learned  well  enough  to 
cease  supervised  training.  The  parent's  work  in  this  subsystem  also  extends 
to  a  "critical  examination"  of  the  child  to  identify  any  special  training 
requirements.  For  example:  Is  the  child  so  short  that  wooden  blocks  should 
be  attached  to  the  pedals?  Has  this  child  shown  any  special  problem  in 
balancing  in  the  past? 

Next,  the  parent  and  child  move  into  the  arena  of  the  Emplacement 
subsystem.  First  of  all,  they  obtain  a  bicycle.  Although  the  economics  of 
that  step  probably  fall  only  on  the  parent,  the  child  very  likely  has  some 
input  to  the  decision  as  to  exactly  what  bike  will  be  had.  If  nothing  else,  the 
choice  of  color  might  be  up  to  the  child.  Training  wheels  might  be  purchased 
at  this  time.  Then,  a  location  for  the  training  has  to  be  chosen.  Here  again 
the  parent  should  take  charge,  since  the  parent  presumably  is  much  better 
aware  of  the  characteristics  of  a  good  practice  site  (traffic-free,  level,  paved, 
no  obstructions,  etc.)  than  is  the  child.  It  would  be  a  rare  parent  indeed 
who  would  entertain  the  possibility  of  constructing  such  a  site.  Instead, 
every  effort  would  be  made  to  find  a  suitable,  existing  site  as  close  to  home  as 
possible. 


-24- 


The  parent  and  child  operate  the  Logistics  subsystem  incidental  to  each 
training  session.  The  parent  should  be  responsible  for  handling  or  at  least 
reviewing  the  more  critical  work  required:  seeing  to  it,  for  example,  that  the 
bike's  nuts  and  bolts  are  secure,  its  tires  properly  inflated,  etc.  Parent  and 
child  likely  will  jointly  inspect  the  practice  site,  finding  and  removing  any  bits 
of  glass  or  other  debris  that  could  degrade  safety  or  otherwise  interfere  with 
learning.  The  parent  might  insist  that  the  child  take  responsibility  for  giving 
the  bike  a  pre-ride  cleaning. 

Finally,  there  is  the  Delivery  of  learning.  The  parent  explains  to  the 
child  the  operations  required  for  bike  riding,  demonstrates  these  operations  to 
clarify  them  in  the  child's  mind,  assists  the  chiles  first  several  rides  by 
holding  on  to  the  bike  to  secure  it,  coaches  the  child  to  guide  him  or  her 
toward  increasingly  better  operation,  and  finally  releases  the  bike  allowing  the 
child  to  practice  unaided  riding.  From  time  to  time,  the  parent  will  have  to 
disentangle  the  fallen  child  from  the  bike,  help  to  alleviate  the  physical  and 
emotic  aal  pain  created  by  the  fall,  boost  the  child's  aspirations  and  resolve, 
and  generally  counsel  the  child  through  the  difficult  trials  of  the  learning 
experience.  Notwithstanding  all  of  this  parental  involvement,  the  child  clearly 
plays  the  key  role  in  Delivery.  It  is  the  child  who  hears  the  explanations, 
6ees  the  demonstrations,  controls  his  or  her  muscles  to  move  and  steer  the 
bike,  contacts  the  ground,  gets  back  up,  dusts  off  and  starts  all  over  again. 
The  child  is  the  one  living  through  the  sensory  activities.  The  sole 
determinant  of  the  success  or  failure  of  training  is  the  child's  ultimate  ability 
to  ride. 


EXAMPLE  2.  A  Large  Training  System 

Next,  consider  briefly  the  training  system  represented  by  a  large, 
private  University.  There  can  be  no  doubt  that  this  is  a  training  -system, 
since  every  University  (officially  at  least)  dedicates  itself  to  learning, 
implements  numerous  sensory  learning  activities,  and  confers  certificates  of 
learning  accomplishment.  Many  Universities  undeniably  pursue  other  functions 
as  well  (research,  football,  etc.),  but  at  its  heart  the  essential  elements  of  a 
training  system  are  present. 

The  University's  key  Command  decision,  viz.,  the  identification  of 
learning  needs,  is  handled  by  the  administrative  staff,  typically  with  the 
advice  of  the  faculty.  Note  that  their  true  purpose  in  undertaking  this  effort 
is  to  ascertain  the  needs  that  exist  in  the  real  world  beyond  the  University's 
walls.  The  administrators  should  ask  themselves:  What  learning  achievements 
are  in  demand  out  there,  by  society  at  large,  potential  employers,  and 
potential  learners?  The  University  functions  in  a  highly  competitive, 
service-oriented  marketplace.  It  may  cease  functioning  if  it  fails  to  anticipate 
and  cater  to  the  learning  needs  perceived  by  the  people  willing  to  pay  for  that 
service. 
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The  Design  subsystem's  principal  operators  are  the  department  heads. 
Presumably,  they  take  their  cues  from  the  needs  identified  by  the 
administration,  filtered  perhaps  by  their  appreciation  of  the  nuances  of 
"marketability"  of  the  skills  and  knowledge  available  in  their  own  disciplines. 
Presumably,  too,  they  seek  the  advice  of  the  members  of  their  departments 
who  may  be  a  bit  more  in  tune  with  the  outside  world  than  are  the  (typically) 
older  and  more  academically  entrenched  chairmen.  Ideally,  the  Design  work 
would  include  interviews  with  real-world  practitioners  of  each  discipline,  to 
benefit  from  their  insights  concerning  the  specific  learning  achievements  that 
really  are  needed. 

In  universities,  the  Enabling  subsystem  usually  is  distributed  among  the 
individual  faculty  members.  Their  initial  acceptance  into  the  university  system 
comes  only  after  they  have  shown  proof  that  they  possess  certain  basic  skills 
in  their  chosen  fields  of  interest  and  in  techniques  of  instruction.  Usually, 
this  "proof"  is  in  the  form  of  learning  certificates  (diplomae)  awarded  to  them 
by  other  universities,  and  of  endorsements  by  other  academicians.  Once 
accepted,  they  are  assigned  certain  duties  as  facilitators,  acquainted  with  the 
relevant  curricula,  and  then  pretty  much  are  left  on  their  own  to  get  ready. 
Faculty  members  usually  have  considerable  latitude  in  adapting  the  curricula  to 
their  own  styles  and  tastes,  in  choosing  the  particular  resource  material  to 
which  the  learners  will  be  exposed,  in  planning  the  details  of  learning 
experiences,  and  in  otherwise  "fleshing  out"  the  curricula  in  preparation  for 
Delivery.  A  university's  Design  subsystem  tends  to  produce  skeletal 
curricula,  relying  on  the  Enabling  process  to  add  the  essential  substance  to 
the  structure.  Thus,  the  same  curriculum,  in  the  hands  of  different  faculty 
members,  can  involve  significantly  different  learning  activities  and  produce 
significantly  different  degrees  of  learning  effectiveness.  At  least  in  this 
regard,  universities  tend  to  differ  from  military  training  systems,  whose 
Design  subsystems  usually  produce  fairly  detailed  curricula  and  whose 
facilitators  have  considerably  less  leeway  in  modifying  the  curricula. 

The  university's  Emplacement  subsystem  for  the  most  part  is 
prefabricated.  It  is  presupposed  that,  whatever  the  learning  needs  and 
objectives  might  be,  much  the  same  kinds  of  learning  activities  will  be 
conducted  and  much  the  same  kind  of  facilities  and  equipment  will  be 
applicable.  Universities  thus  expend  considerable  funds  to  provide 
general-purpose  classrooms,  libraries,  laboratories,  auditoria,  media  centers, 
and  the  like.  Undeniably,  this  impacts  on  the  Design  and  Delivery  subsystems 
since  it  induces  a  strong  pressure  to  use  these  facilities,  and  to  shape  the 
learning  activities  to  conform  to  the  available  Emplacement  resources.  To  the 
extent  that  military  training  systems  have  copied  the  university's  predilection 
for  expensive,  fixed  facilities,  a  predisposition  against  individualized, 
work-site  training  of  soldiers  has  been  created,  even  though  work-site  training 
might  be  more  effective  in  many  cases. 

The  university's  Logistics  subsystem  also  tends  to  be  general-purpose. 
Housing,  feeding,  and  recreation  of  learners,  facilitators,  logistics  supporters, 
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etc.,  are  accomplished  on  a  mass  scale,  usually  in  that  efficient  but  colorless 
style  known  as  "institutional."  Physical  maintenance  of  facilities  and  equipment 
is  similarly  organized  and  implemented  as  a  routinized,  impersonal  process. 

As  in  any  training  system,  it  is  in  the  university's  Delivery  subsystem 
that  the  learner  does  his  or  her  work.  To  a  great  degree,  the  learner's 
sensory  activities  consist  of  those  that  are  traditional  in  an  academic  setting: 
listening  to  lectures,  viewing  an  occasional  film,  reading  books,  writing 
papers,  participating  in  laboratory  experiments,  etc.  The  more  fortunate 
learner  might  occasionally  have  an  opportunity  to  participate  in  role-playing 
experiences,  small  interactive  study  groups,  self-paced  computer-assisted 
activities,  and  a  generally  richer  environment  with  more  active  sensory 
experiences  to  enhance  learning.  As  a  subclass  of  training  systems,  however, 
the  university  may  be  among  the  slowest  to  adopt  modern  technology  to  its 
learning  and  learning-helping  activities. 


The  authors  hope  that  these  two  examples  have  convinced  the  reader  that 
the  postulated  six  subsystems  represent  a  reasonable  and  useful  general  model 
for  any  training  system.  It  now  remains  to  apply  that  model  to  construct  a 
general  representation  of  training  system  performance,  and  to  extend  that 
representation  to  particular  measurement  and  design  applications. 

B .  Taxonomies  of  Training  System  Performance 

The  authors  have  asserted  that  every  training  system  includes  subsystems 
t>  at  have  been  labeled  Command,  Design,  Enabling,  Emplacement,  Logistics, 
and  Delivery,  The  level  of  effort,  the  complexity,  the  time,  the  cost,  and 
other  attributes  of  the  work  carried  out  by  those  subsystems  no  doubt  vary 
widely  from  one  training  system  to  another,  as  demonstrated  by  the  examples 
above.  But  even  though  the  amount  of  work  needed  varies,  the  performance 
requirements  are  stable.  Each  of  the  training  subsystems  exists  because  it 
makes  some  essential  contribution  to  the  master  functions  of  learning  and 
helping-to-learn.  It  doesn't  matter  whether  the  training  system  is  large  or 
small,  simple  or  complex.  The  subsystem  contributions  are  needed  in  every 
case. 


What  are  the  contributions  needed  from  each  of  the  six  subsystems?  What 
performance  requirements  must  they  satisfy  if  the  total  training  system  is  to 
achieve  its  purposes  of  learning  and  helping-to-learn? 

The  performance  requirements  of  each  training  subsystem  can  be 
identified  by  applying  the  systems  taxonomization  guidelines  (see  Figure  2)  to 
define  the  taxa  or  categories  of  potentiality,  process,  and  product  which 
describe  the  subsystem  and  hence  the  populations  to  which  it  belongs.  For 
example,  one  can  examine  training  Command  subsystems  in  general  and  ask: 
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1.  What  capabilities  are  to  be  provided 
by  Command?  What  contribution  is 

Command  supposed  to  make  to  the  (Potentialities) 

abilities  the  system  needs  to  accom¬ 
plish  learning  and  helping-to-learn? 

2.  What  activities  are  supposed  to  be 
carried  out  by  Command?  How  is 

Command  supposed  to  contribute  to  the  (Processes) 

procedures  the  system  employs  to 
accomplish  learning  and  helping-to- 
learn? 

3.  What  goods  or  services  are  supposed 
to  be  produced  by  Command?  What 

contribution  is  Command  supposed  to  (Products) 

make  to  creating  the  products  the 
system  needs  to  accomplish  learning 
and  helping-to-learn? 

These  and  similar  questions  will  lead  the  analyst  to  discover  the 
Objectives-level  performance  requirements  of  the  Command  subsystem,  i.e.,  the 
fundamental  "what"  of  Command  performance  in  terms  of  potentialities  to  be 
possessed,  processes  to  be  undertaken,  and  products  to  be  delivered. 

Each  Objectives-level  requirement  car.  be  analyzed  in  greater  depth,  again 
using  the  taxonomization  guidelines,  to  determine  "why"  it  is  needed.  That 
step  will  disclose  the  Functional  Purpose-level  taxa  to  which  Command  belongs. 
These  in  turn  can  be  analyzed  to  determine  "how"  the  purposes  are  to  be 
fulfilled,  producing  Command's  Characteristics-level  performance  taxa.  This 
same  analytic  approach  can  be  taken  to  derive  the  performance  taxa  for 
Design,  Emplacement,  Logistics,  Enabling  and  Delivery.  Taken  together,  the 
six  taxonomies  will  cover  all  of  the  performance  requirements  of  the  total 
generalized  training  system. 

The  first  stages  of  the  six  taxonomies  have  been  derived  and  are 
displayed  in  Figure  6.  The  items  listed  are  the  Objectives-level  taxa,  i.e., 
the  "what"  of  performance  requirements  for  each  training  subsystem.  Those 
taxa  could  be  applied  to  specify  and/or  evaluate  the  design  of  a  given  training 
system  at  the  conceptual  stage  of  development.  In  effect,  the  Objectives-level 
taxa  constitute  a  check  list  for  the  training  designer/evaluator.  The 
designer /evaluator  can  review  each  taxon  and  ask,  "Is  this  particular 
capability,  procedure,  or  deliverable  product  relevant  to  my  training 
application?  If  yes,  has  my  training  system  concept  made  provision  for 
including  the  capability,  procedure  or  product?  If  no,  how  should  my  concept 
be  modified  to  correct  that  deficiency?"  Note,  however,  that  the 
Objectives-level  taxa  say  nothing  about  the  particular  purposes  to  which  the 
system  in  question  will  put  each  capability,  procedure^  and  product,  and 
certainly  also  say  nothing  about  the  design  details  needed  to  insure  that  those 
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Figure  6.  Objectives-Level  Performance  Taxa  of  Training  Subsystems 


purposes  will  be  accomplished.  That  is,  the  "what"  of  performance  says 
nothing  about  the  "why"  or  "how"  of  performance.  Taxa  must  be  identified  on 
the  functional  purposes-  and  characteristics-levels  in  order  to  obtain  inputs 
relevant  to  "why"  and  "how"  issues  concerning  design  and  evaluation. 

To  date,  taxonomies  complete  on  all  three  levels  have  been  compiled  for 
the  Design  and  Enabling  subsystems.  These  two  were  singled  out  to  initiate 
training  system  taxonomization  for  two  reasons.  First,  they  are  the  most 
extensively  studied  and  best  documented  of  the  training  subsystems. 
Procedural  manuals  and  research  reports  abound  that  deal  with  both  civilian 
and  military  applications  of  instructional  system  Design  and  Enabling  (the 
Liter,  typically  under  the  title  "Instructor  Training").  Representative 
documents  from  that  literature  base  were  reviewed  in  depth  in  preparation  for 
the  present  work,  which  then  proceeded  in  a  less  speculative  atmosphere  than 
might  have  been  the  case  if  other  subsystems  had  been  selected.  It  might  be 
added,  in  this  same  context,  that  the  authors'  have  substantial  prior 
experience  in  training  systems  Design  and  Enabling  applications. 

The  second  major  reason  for  addressing  these  two  subsystems  is  that  they 
are  probably  of  greatest  concern  to  training  systems  at  early  stages  of 
development.  Two  important,  emerging  Army  training  systems  are  at  such 
stages,  namely,  the  Bradley  Infantry  Fighting  Vehicle  Training  System 
(BIFVTS)  and  the  new  9  mm  handgun  training  system.  The  developers  of 
those  systems  conceivably  could  benefit  from  a  specification  of  Design  and 
Enabling  performance  requirements.  Figures  7  and  8,  respectively,  present 
complete  taxonomies  for  the  Design  and  Enabling  training  subsystems. 

The  taxa  contained  in  Figures  7  and  8  represent  the  basic  building  blocks 
for  both  design  and  evaluation  of  the  two  training  subsystems.  Each  taxon  is 
a  performance-oriented  design  requirement.  The  subsystem  designer  examines 
a  particular  taxon  and  asks,  "How  can  I  best  insure  that  this  requirement  will 
be  satisfied  by  the  subsystem  I  am  building?"  The  evaluator  looks 
independently  at  the  same  taxon  and  asks,  "How  can  I  tell  whether  this 
particular  subsystem  satisfies  that  requirement?"  Answers  to  the  designer's 
question  produce  system  specifications.  Answers  to  the  evaluator's  question 
lead  to  measures  of  system  performance/effectiveness. 

A  decimal  indexing  code  was  established  to  facilitate  the  categorization  of 
all  three  levels  of  system  description.  Each  contextual  aspect  was  assigned  a 
unique  number,  as  follows: 


1.0  Performance  Potentialities 
2.0  Performance  Processes 
3.0  Performance  Products 
JA-C'. 


At  the  Objectives  Level,  the  population  categories  is  assigned  successive 
numbers  in  the  first  decimal  position.  For  example,'  ai-the  Objectives  -Level, 
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1.1.2 

Stating  the  ultimate 

2.1.2 

Determine  the  sequence 

constitutes  the  job  to  br 

intended  outcomes  of 

and  stimuli  of  the  tasks 

trained 

learning 

2.1.3 

Determine  the  level  of 

3.1.2 

Provide  a  complete 

1.1.1 

Identifying  the  rala- 

skill  or  performance 

specification  of  the  or- 

tive  importance  of  in- 

deemed  adequate  for  the 

ganJietlon  of  the  job 

tended  outcomes 

job 

3.1.3 

Provide  a  complete 

1.1.4 

Eatabiishing  a  basis 

specification  of  what 

for  specifying  learning 

2.2.1 

Determine  needs  for 

constitutes  job  adequacy 

objectives 

training  as  a  function 

of  task 

3.2.1 

Define  exactly  what  the 

1.2.1 

Stating  what  the  learners 

2.2.2 

Allocate  training 

learners  are  expected 

will  be  able  to  do  or  how 

development  and 

to  achieve  a>  ■  result 

they  will  be  expected  to 

delivery  resources 

of  the  training 

behave  after  training 

3.2.2 

Provide  an  organiia- 

1.2.2 

Providing  a  basis  for  ob- 

2.3.1 

Identify  the  information 

tional  framework  for 

jedively  assessing 

processing  entailed  In 

the  training 

learner  performance 

the  task 

2.3.2 

Classify  the  tasks  in 

3.3.1 

Provide  complete  sped- 

1.3.1 

Identify!*.*  prerequisites 

terms  of  the  domains  of 

flcation  of  the  Inatruc- 

to  learning 

learning  involved 

tional  content  for  each 

l.J. 2 

Providi.-ijr  *  basis  for 

2.3.3 

Identify  the  conditions 

objective 

objectively  assessing 

for  learning  the  task 

3.3.2 

Provide  complete  sped- 

*  car.  hdafp  learner's 

flcation  of  the  instruc* 

suitabii  ty 

2.4.1 

Identify  Instructional 

tional  procedures  for 

events/activities  required 

each  objective 

1.3  1 

Identifying  Interactions/ 

2.4.2 

Identify  strategic  instruc- 

J.J.J 

Provide  complete  epeci- 

dependence  among 

tional  requirements 

flcation  of  the  instruc- 

objectives 

2.4.3 

Identify  tactical  inatruc- 

tional  circumstances 

l.J. 4 

Classifying  objectives 

tional  requirements 

for  each  objective 

in  terms  of  domains 

2.4.4 

Identify  learner  ateees- 

of  learning 

tnent  requirements 

3.4.1 

Provide  essentia! 

1.3.5 

Identifying  component 

resources  to  learners 

etepa  or  proceaaea 

2.5.1 

Asaese  relevance  and 

3.4.2 

Provide  essentia!  guid- 

within  the  objectives 

applicability  of  existing 

a  nee  to  facilitators 

instruction  to  present 

3.4.3 

Provide  eeaential  sped- 

1.4.1 

Selecting  stimuli  for 

requirements 

ficetlona  to  facilities 

aach  objective 

2.5.2 

Aateee  effectiveness/ 

devalopera 

1.4.2 

Selecting  instructional 

efficiency  of  exieting 

3.4.4 

Provide  eeaential 

•ettlng  for  each 

instruction  in  light  of 

•upport  to  training  ays- 

objective 

etete-of-art  methods/ 

tern  administrators 

1.4.3 

Defining  inetructiona! 

media 

sequence 

3.5.1 

Acquire  qualified 

2.6.1 

Adapt  relevant  and  appli- 

candidate  learners 

1.3.1 

Specifying  preparation 

cable  instruction,  where 

3.5.2 

Tailor  Instruction  to 

procedures 

poesible,  to  satisfy  pre- 

teamara 

1.5.2 

Specifying  presentation 

sent  instructions! 

3.5.3 

Meaaura  learner's 

procedures 

requirements 

achievements 

1.5.3 

Specifying  application 

2.6.2 

Create  instruction*  where 

procedures 

necessary,  to  meet  unset- 

3.6.1 

Provide  justification  for 

1.5.4 

Specifying  (learner) 

itfieri  instructional 

•pacific  features  of 

evaluation  procedures 

rcqulremen  ta 

inetructiona)  design 

3.6.2 

Demonstrate  validity 

1.4.1 

Validating /revising 

2.7.1 

Guide  the  ongoing  analysis 

•nd  utility  of  the 

the  bases  for  the 

and  assembly  of  Inetructton 

training 

developing  curriculum 

2.7.2 

Verify  the  suitability  of 

1.4.2 

Validating /revising 

the  aeeembled  inetructton 

ihe  developing  curri- 

for  delivery 

culum  itaelf 

Figure  1.  Taxonomy  of  Design  Subsystem  Performance 
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ritARACTF.Biynr? 


performance 

POTENTIALITIES 


PERFORMANCE 

PROCESSES 


PERFORMANCE 

PRODUCTS 


1.1.11 

Identifying  types  of 
achievements  relevant 
to  the  intended  job 

1.1. 1.2 

Analysing  schievements 
to  determine  suitability 
for  training 

1. 1.2.1 

Determining  necessary 
levels  of  achievement 

1. 1.2.2 

Determing  existing 
levels  of  achievement 

1.1.2.) 

Discerning  discrepan¬ 
cies  between  existing 
and  necessary  levels 

1.1.3.) 

Assessing  goal 

Importance 

1.1.S.2 

Insuring  svailability 
of  s  relative  (numeric) 
goal  ranking  scheme 

1.1.4.  1 

Insuring  svailability 
of  a  hierarchy /taxonomy 
of  objectives 

1. 1.4.2 

Ststing  goals  in  terms 
that  imply  behaviors 

1.2. 1.1 

Defining  the  perfor¬ 
mance  sction 

1.2. 1.2 

Defining  the  perfor¬ 
mance  conditions 

1.2. 1.3 

Defining  the  perfor¬ 
mance  standard 

1.2.2. 1 

Developing  pretest 
items 

1.2. 2.3 

Developing  post  test 
items 

1.1. 1.1 

Reveal)  g  present  con¬ 
ditions  of  learning 

0 .e.  .  that  which 
learners  must  recall) 

1.3. 1.2 

Identifying  essential 
prerequisites  (capabili¬ 
ties  Incorporated  within 
the  objective) 

1.3. 1.3 

Identifying  supportive 
prerequisites  (capabili¬ 
ties  facilitating 
achievement  of  objectives) 

1.3.2. 1 

Developing  entry  test 
Items 

1.3.3. 1 

Discerning  independence 
between  objectives 

1.3. 3. 2 

Discerning  dependency 
between  objectives 

1.3. 1. 3 

Discerning  supportive 
relationship  between 
objectives 

1.3. 4.1 

Identifying  information 
type  objectives 

1.3. 4. 2 

Identifying  mental 
skills  type  objectives 

2.1.1.) 

Review  job  specification/ 
description 

2. 1.1. 2 

Interview  job/syctem 
designers 

2. 

1.1.3 

Observe  job  incumbents 

2. 1.1. 4 

Interview  job  incumbents 

2. 1.1. 5 

Prepare  tentative  task 
list 

2. I. 1.4 

Authenticate  tenatfve  task 
litil  using  representative 
job  designers/incumbents 

2. 1.1.7 

Validate  task  list  using 
other  representatives 

2. 1.2.1 

Review  job  specification/ 
description 

2. 1.2. 2 

Interview  job/system 
designers 

2. 

1.2.3 

Observe  job  incumbents 

2. I. 2. 4 

Interview  job  incumbents 

2. 1.2. 5 

Prepare  tentative  task 
flowchart 

2. 1.2.4 

Prepare  tentative  task 
stimuli  specifications 

2. 1.2. 7 

Authenticate  tentative 
flowchart  and  stimuli 
specifications  using 
representative  Job 
designers/incumbents 

2. 1.2.8 

Validate  flowchart  and 
stimuli  specifications 
using  other  representatives 

2. 1.3.1 

Review  Job  specification/ 
description 

2.1 

1.3.2 

Interview  job/syst em 
designers 

2.1.3.) 

Interview  job  supervisors/ 
cc-mmanders 

2. I. 3. 4 

Obaerve/mterview  *high*- 
rated  inevimbents 

2.1 

3.5 

Observe/interview  •low#- 
rated  incumbent* 

2.J.3.4 

Prepare  tentative  skill/ 
performance  specifications 

2. 1.3.7 

Authenticate  performance 
specifications  using 
representative  designers/ 
aupervisors/incumbents 

2. 1.3. 8 

Validate  performance 
specifications  using  other 
representatives 

2. 2. 1.1 

Develop  training  needs 
criteria  in  concert  with 
representative  system 
usera/designers 

2. 2. 1.2 

Rate  tasks  in  accordance 
with  the  specified 
criteria 

2. 2. 2.1 

Determine  degree  of 
learning  difficulty  for 
each  task  requiring 
training 

2  2. 2. 2 

Determine  degree  of 
Instruction  needed  for 
each  such  task 

3. 1.1.1 

Specification  of  mental 
processes  to  the  level  of 
binary  decisions 

3.1. 1.2 

Specification  of  physical 
processes  to  be  lev?]  of 
discrete  actions 

3. 1.1. 3 

Specification  of  all 
process  stimuli 

3. 1.1. 4 

Specification  of  condi¬ 
tions  under  which  pro¬ 
cesses  are  to  take 
place 

3. 1.2.1 

Operational  sequence 
diagrams 

3. 1.2. 2 

Decision  flowcharts 

3. 1.2. 3 

Specification  of 
rules  and  regulations 

3.1.3. 1 

Performance  accuracy 
specification 

3. 1.3. 2 

Performance  speed 
Specification 

3. 1.3. 3 

Performance  volume 
specification 

3. 1.3. 4 

Performance  duration 
specification 

3. 2. 1.1 

Specification  of  the 
basic  capabilities  the 
learners  are  to  acquire 

3.2. 1.2 

Specification  of  an  ob 
servable  action  the 
learners  will  execute  •  > 
demonstrate  each 
capability 

3. 2. 1.3 

Specification  of  a  mea¬ 
surable  object  that  will 
result  from  the  action 

3.2. 1.4 

Specification  pf  the 
circumstances  under 
which  the  learners  will 
execute  the  action 

3. 2. 1.5 

Specification  o.  ny  tools 
or  equipment  the  learners 
will  uae  to  execute  the 
action 

3. 2. 1.4 

Specification  of  any 
eonatrainta  imposed  upon 
the  learners'  execution 
of  the  sction 

3. 2. 1.7 

Specification  of  the  cri¬ 
teria  the  resulting  object 
must  satisfy  in  order  to 
demonstrate  acceptable 
achievement  of  the  basic  i 
capability 

3.2.2. 1 

Specification  of  the  ' 

essential  abilities  includ¬ 
ed  within  each  perfr, 
mane*  objective 

3. 2. 2. 2 

Specification  of  the 
supportive  ablUttea  to 
be  included  in  Instruc¬ 
tion  for  each  objective 

CHARACTERISTICS  (Cont'd) 


PERFORMANCE 

POTENTIALITIES 

performance 

PROCESSES 

performance 

PRODUCTS 

3.2.23 

Specification  ol  the  a e- 

1.3.4.  ) 

Identifying  physical 

quence  of  Instruction  in 

■*slls  type  objective* 

2. 3. 1.1 

Analyse  the  decision 

essentia]  and  supportive 

1 .3.4.4 

Identifying  attitude 

Dow  to  reduce  pro- 

abilities  to  be  prepared 

type  objective* 

cessing  to  a  sequence 

for  each  objective 

of  binary  decisions 

3. 2. 2. 4 

Specification  of  the  *e- 

1.3.5  1 

Ider, tifying  overt  steps 

2. 3. 1.2 

Analyse  the  decision 

quence  of  objectives  to 

1  .3.5.2 

Identifying  covert 

levels  in  terms  of 

be  following  in  the 

•  tep« 

nature,  complexity,  and 

instruction 

1  1.5. i 

Identifying  unconscious 

time 

•  tv  ps 

3. 3. 1.1 

Specification  of  the 

2. 3. 2.1 

Identify  knowledge  to  be 

objective 

4.1.1 

Defining  propositions , 

acquired 

3. 3. 1.2 

Specification  of  usumr  i 

nimei,  facts,  etc.. 

2. 3. 2. 2 

Identify  mental  skills  to 

learner  prerequisites 

relevant  to  informa 

be  acquired 

3.3. 1.3 

Itemization  of  informa- 

tior  objectives 

23.2.3 

Identify  physical  skills 

tion  topics 

1.41.2 

Defining  concepta.  rules. 

to  be  acquired 

3. 3. 1.4 

Itemization  of  mental 

algorithms,  etc.,  rele- 

2 .  3 .  .\  4 

Identify  attitudes  to  be 

skills  steps 

vant  to  mental  skills 

acquired 

3. 3. 1.5 

Itemization  of  physical 

ob-ect  •  ves 

•kills  steps 

;  .4.  !  .3 

Defning  movement*, 

2.3.3. 1 

Analyte  information 

3. 3. 1.6 

Itemization  of 

timings,  actions,  etc.. 

processing  requirements 

attitudinal  components 

relevant  to  physical 

to  identify  prerequisite 

1.3. 1.7 

Itemization  of  practice 

skill*  objectives 

knowledge  expected  of 

exercises 

1.4.!. A 

Defining  values,  choices, 

learners 

3. 1.1. 8 

Itemization  of  test 

etc. ,  relevant  to 

2. 3. 3.2 

Analyte  those  require- 

problems /exercises 

attitudinal  objectives 

ments  to  identify  pre- 

requisite  mental  skills 

1.4. 2.1 

Assessing  applicability 

2. 3. 3.3 

Analyte  the  physical 

3.3. 2.1 

Specification  of  pro- 

of  Job  Performance  Aids 

requirements  to  identify 

cedures  for  preparing 

(JFA.) 

prerequisite  motor  skills 

learner,  to  achieve  the 

1 . 4  .  ? .  2 

Assessing  applicability 

2.3  3.4 

Analyte  the  attitudinal 

objective 

of  Self-Teaching  Export- 

requirements  to  Identify 

3. 3. 2. 2 

Specification  of  how  the 

able  P.cksges  (STEP.) 

prerequisite  values 

Instructional  content 

1 .4.2,3 

Aiieuin,  applicability 

toptc«/,tep,  will  be 

ol  Formal  on-the-Job 

2. 4. 1.1 

Select  the  types  of 

presented 

Training  (FOIT) 

evenls/activities  required 

3. 3.2.3 

Specification  of  how  the 

1.4. 2. 4 

Assessing  apj;  lie  ability 

to  prepare  learners  to 

lesrners  will  practice/ 

of  !nsta)latior.  Support 

learn  the  v.rioua  talk. 

apply  the  Instructions] 

School  (1SS) 

.'.4.1.2 

Select  the  types  of 

content 

1.4. 2. 5 

Assessing  applicability 

events/activities  required 

3. 3. 2. 4 

Specification  of  how  the 

of  Reside..!  School  (RS) 

to  present / demonstrate 

learners  will  be  tested 

the  tasks 

on  the  Instructional  j 

1.4. 1.1 

Sequencing  in  depen- 

2. 4. 1.3 

Select  the  types  of 

content 

dency  order 

events /activities  required 

3. 3. 2. 5 

Specification  of  proce- 

1 .4. 3.2 

Sequencing  In  supportive 

for  leirner,  to  practice/ 

dure,  for  dealing  with 

iclationship  (efficiency) 

apply  the  tasks 

learners  who  fail  to 

order 

2.4. 1 .4 

Select  the  types  of 

demonstrate  achievement 

events/activities  required 

of  the  objective 

1.5. 1.1 

Defining  facilitator 

to  evaluate  learners' 

preparation  activities 

performance 

3. 3. 3.1 

Specification  of  the 

1.5. 1.2 

Defining  facilities 

location  at  which  each 

preparation  requirements 

2. 4. 2.1 

Analyse  events/activities 

instructional  event  will 

1.5. 1.1 

Defining  attention- 

to  establish  their  sequence 

take  place 

gaining  procedures 

2. 4. 2. 2 

Anslyce  events/actlvities 

3. 3. 3. 2 

Specification  of  the 

1.51.4 

Defining  objective- 

to  establish  their 

equipment  needed 

Informing  procedures 

delivery  media 

3. 3.3.3 

Specification  of  speria’ 

1.5.15 

Defining  procedures  for 

location  /equipment 

stimulating  learners' 

2. 4. 3.1 

Analyse  events  (activities 

•et-up/configur  ation 

recall  of  prerequisites 

to  establish  their 

requirements 

management  procedures 

3. 3. 3. 4 

Specification  of  tnptrur- 

1.5.2.. 

Defining  presentation 

2.4.32 

Analyse  events/activities 

tlonat  personnel  needed 

media /media 

to  establish  their 

3. 3. 3. 5 

Specification  of  time  j 

alternatives 

resource  requirements 

and  scheduls 

1.5. 2. 2 

Defining  procedures  and 

requirements 

requirements  for  placing 

2. 4. 4.1 

Analyse  taaka  to  identify 

emphasis 

3.4. 1.1 

Text, 

1.5. 2. 3 

Defining  procedures  for 

for  selecting/aceeptlng 

3. 4. 1.2 

Workbook, 

enhancing  learners' 

learners 

3.4. 1.3 

Study  guide. 

comprenension  and 

3. 4. 1.4 

Lists  of  references 

retention 

3. 4. 1.5 

Schedule  of  Instruction 

Figure  7  (Continued) 
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CHARACTERISTICS  (root'd) 


PERFORMANCE 
POTEh  T1AUTIES 


PERFORMANCE 

PROCESSES 


PERFORMANCE 

PRODUCTS 


2. 4, 4. 2  Analyie  tasks  to  identify 

assessment  requirements 

for  providing  remedial 

1.5.3.]  Defining  procedures  for 

instruction 

3.4.2. 1  Sets  of  lesson  plans 

enhancing  learners' 

2. 4.4.3  Analyse  tasks  to  identify 

3, 4.2.2  Sets  of  test 

involvement/ 

assessment  requirements 

apecificationa 

pzrticipition 

for  accelerating /tailoring 

3.4.2. 3  Lists  of  facihuior 

1.-.3.2  Pe/inir.g  procedures  for 

learners'  progress  through 

re ferences 

•hewing  performance 

the  instruction 

3. 4. 2. 4  Specifications  of  facill- 

from  1 earners 

2, 4. 4. 4  Analyse  tasks  to  identify 

tator  prerequisites 

1.5. 3. 3  Defining  procedures  for 

assessment  requirements 

providing  learning 

for  advancing/deUylng 

3.4. 3.1  Specifications  of 

guidance  to  learners 

learners'  progress  through 

physical  requirements 

1.5. 3. 4  Defining  procedures  for 

the  instruction 

for  instructional  sites 

providing  learning 

2. 4. 4. 5  Analyze  tasks  to  identify 

3.4. 1.2  Specifications  for 

feedback  to  learners 

audiovisuals 

1.5. 3. 5  Defining  procedures  for 

for  certifying/decertifying 

3. 4. 3. 3  Specifications  for 

assessing  sufficiency 

learners  as  job  performers 

training  equipment  and 

of  practice 

supplies 

2.S.1.1  Specify  criteria  of  rele- 

1.5. 4.1  Defining  procedures  to 

vance  and  applicability 

3. 4.4.1  Detailed  outline  of 

assess  the  proper 

2.5. 1.2  Search  for  existing 

instructional  objectives. 

domains  of  learning 

Instruction  of  potential 

content,  procedures. 

1.5. 4, 2  Defining  procedures  to 

relevance  and  applicability 

and  schedule 

•  •sets  the  appropriate 

2. 5. 1.3  Access  and  review  existing 

3. 4. 4. 2  Specifications  of  learner 

act  ion /behavior 

instruction  deemed  to  be  of 

prerequisites 

i.5.4.3  Defining  procedures  to 

potential  relevance  and 

3. 4. 4. 3  Summaries  of  personnel 

insure  assessment 

applicability 

requirements 

reliability 

2. 5. 1.4  Rate  relevance/apphc- 

3. 4. 4. 4  Summaries  of  facilities 

ability  of  instruction 

requirements 

3 . 6 . 1 . 1  Entry-level  testing  of 

2. 5. 1.5  Select  existing  instruction 

3. 4. 4. 5  Documentation  of  all 

'i 

representative  ear-.di- 

warranting  application  to 

activities  during  the 

C 

date  learners 

present  instructional 

process 

1.6. 1.2  Post  -  training  testing 

requirements 

o 

of  representative  joo 

p 

incumbents 

2.5.2. 1  Identify  instructional 

3. 5. 1.1  Measures  of  prerequisite 

£ 

eventa/activities  imple- 

abilities 

Ui 

1.6?  1  Obtaining  error  data 

mented  by  technologically 

3.5.1  .Z  Measure,  application 

C 

1.6.4  2  Obtaining  design  feed- 

outmoded  methods/media 

procedures 

< 

back 

2. 5.2, 2  Assess  degree  of  ineffi- 

3. 5. 1,3  Standards  of  candidate 

K 

1.6. 2. 3  Correlating  post-test 

ciency  /  ineffectiveness 

qualification 

data  with  subsequent 

2.5.2. 3  Assess  cost  of  replacing 

3. 5. 1.4  Specifications  for  deal- 

U 

Job  performance 

outmoded  instruction 

in  g  with  substandard 

2  5.2.4  Formulate  decision  to  use 

candidate! 

or  replace  existing 

Instruction 

3.5.2. 1  Pre-training  measures 

of  intended  abilities 

2. 6. 1.1  Identify  specific  deft- 

3. 5. 2, 2  Measures  application 

cienciea  in  the  events/ 

procedures 

activities  of  the 

3. 5. 1. 3  Standards  of  pre- 

applicable  Instruction 

training  qualification 

2. 6. 1.2  Develop  specific  changes 

3. 5.2.4  Sped fl cation  of  pro- 

to  the  sventa/artlvlties 

cedures  for  dealing 

and  their  prescriptions 

with  qualified  learners 

to  remove  the  deficiencies 

3. 5. 3.1  Poat-trslning  measures 

2. 6.2.1  Identify  specific  objec- 

of  intended  abilities 

lives  for  which  new 

3.5. 3.2  Measures  application 

tnetruction  i»  needed 

procedures 

2. 6. 2. 2  Analyze  the  objectives 

3. 5. 3. 3  Standards  of  post- 

to  identify  the  Instruc- 

training  qualification 

tional  events/activities 

3.5.3. 4  Specification  of  pro- 

needed 

cedures  for  dealing  with 

Z.t.Z.  3  Select  implementing 

unqualified  learners 

methods/media 

2.6. 2. 4  Prepare  detailed  prs- 

3. 6. 1.1  Entry  fast  data  derived 

scriptiona  of  the 

from  representative 

method!  end  media 

candidate  leenrters 

Figure  7  (Continued) 


Y-L' 


-34- 


t 


PERFORMANCE 

POTENTIALITIES 


PERFORMANCE 

PROCESSES 


Apply  ■mall  block*  of 
existing  instruction  to 
individual  candidate 
learners 

Apply  small  blocks  of 
new  instruction  to  indi¬ 
vidual  candidate  learners 
Apply  small  blocks  of 
assembled  instruction  to 
small  groups  of  candi¬ 
date  learners 

Conduct  field  testing  of 
the  assembled  instruction 


PERFORMANCE 

PRODUCTS 

3.6. 1.2  Post-test  data  derived 
from  representative 
job  incumbent* 

3. 6. 1.3  Teat  item  error  analyses 
derived  from  tryout 
learnera 

3. 6. 1.4  Inatruction  design  feed¬ 
back  data  obtained  from 
tryout  learnera  and 
facilitator* 

3.6.1.5  Specification  of  revision* 
based  on  these  data 

3.6.2. 1  Teat  item  error  analy¬ 
ses  derived  from  field 

teat  learner* 

3. 6. 2. 2  Correlation*  between 
learners'  post-teat 
rasults  and  subsequent 
job  performance 

3.6. 2. 3  Instruction  design  feed¬ 
back  data  obtained  from 
field  test  learnars 

and  facilitators 

3. 6. 2. 4  Specification  of  final 
revisions  based  on 
these  data 


Figure  7  (Concluded) 


\ ••  V-’SV - V -V.-ur-v’ 


PERFORMANCE 

POTENTIALITIES 


PERFORMANCE 

PROCESSES 


PERFORMANCE 

PRODUCTS 


1.1.1 


1.1.2 


1.1.3 


1.1.4 


1.1.5 

1.1.4 

1.2.1 

1.2.2 

1.2.1 

1.2.4 

1.2.5 

1.1.1 

1.3.2 

1.3.1 


1.3.4 

1.1.5 

1.1.4 

1.1.7 

1.4.1 


1.4.2 

1.4.1 


Providing  day-to-day  2.1 

training  management 
rcaourcea 

Insuring  availability  2.2 

of  instructional 

delivery  expertise 

Insuring  availability 

of  learning  guidance  2.1 

expertise 

Insuring  availability 

of  subject  matter  2.4 

expertise 

Familiarising  facili¬ 
tators  with  the  2.5 

curriculum 

Tailoring  the  curricu¬ 
lum  to  specific 
delivery  applications 


Establishing  a  ached-  2.1.1 

ule  of  learning 

activities 

Establishing  a  condu¬ 
cive  learning  climate  2.1.2 

Monitoring  and  modify¬ 
ing  the  learning 
experience 

Operating  the  training 
facilities  and  equip¬ 
ment  2.2.1 

Implementing  learning 
activities 

Acquiring  and  dissemi¬ 
nating  learning  2.2.2 

resources 


Conveying  learning 
stimuli 

Presenting  information 
Demonstrating  pro¬ 
cedures 

Conducting  learner 
practice 

Conducting  learner 

testa 

Motivating  learners 
to  learn 

Focusing  learners' 
attention  and  interest 
Relating  the  learning 
to  the  learners' 
experience 
Coaching  lasmers 
Feeding-bach  to 
learners 

Facilitating  learner 
self-aaaeasmen  t 
Counseling  learners 

Knowing  the  ingredi¬ 
ents  of  that  which  is 
to  be  learned 
Perfomin  g  /  demon  strat  - 
big  that  which  ia  to  . 
be  learned 
Citing  illustrative 
examples  to  assist 
learning 


2.1.1 


2.1.2 


2.4.1 


2.4.2 


2.4.3 


Analyte  facilitator's  3.1 

ability  to  perform  the 

tasks 

Analyte  facilitator's  3.2 

ability  to  manage /imple¬ 
ment  the  learning 
activities 

Design  activities  to 
correct  facilitator's 
deficiencies 
Design  activities  to 
allow  adaptation  of  the 
curriculum 
Conduct  the  facilita¬ 
tor's  learning  activi¬ 
ties 


Identify  tasks  for  which 
a  given  prospect  could 
credibly  serve  as  a 
facilitator 

Identify  task  performance 
deficiencies  for  a  given 
prospect  which  could  be 
corrected  through  cost- 
effective  training 

Identify  learning  activi¬ 
ties  which  a  given  pros¬ 
pect  could  credibly 
manage /implement 
Identify  deficiencies  in 
learning  activity  manage- 
ment/implementation  for  a 
given  prospect  which 
could  be  corrected 
through  cost-effective 
training 


3.1.1 


1.1.2 


1.2.1 

3.2.2 


Devise  plans  to  Insure 
that  each  facilitator 
achieves  requisite  levels 
of  job  performance  skills 
Devise  plans  to  insure 
that  each  facilitator 
achieves  requisite  levels 
of  instructional  skills 

Devise  plans  to  insure 
that  each  facilitator 
learns  the  curriculum  and 
its  Instructional 
requirements 
Devise  plans  to  Insure 
that  each  facilitator 
learns  the  particular 
requirements  of  a  given 
delivery  application 
Devise  plans  to  insure 
that  each  facilitator 
develops  the  additional 
details  of  procedure  and 
materials  needed  for  a 
given  delivery  applica¬ 
tion 


Qualified  facilitators 
for  given  delivery 
applications 
Curricula  tailored  to 
given  delivery  apph- 
cationa 


Facilitators  able  to 
help  create  a  rich 
learning  environment 
for  the  learners 
Facilitators  able  to 
guide  learners'  inter¬ 
actions  with  the 
environment 

Curricula  relevant  to 
the  learning  needs  ol 
given  learners 
Curricula  geared  toward 
the  experirncea  of 
given  learnrra 


I 


Figure  8.  Taxonomy  of  Enabling  Subsystem  Performance 
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CHARACTERISTICS  I  FUNCTIONAL  PURPOSES  (Cont*d) 


PERFORMANCE 

PERFORMANCE 

POTENTIALITIES 

PROCESSES 

2.5.1 

Facilitators  demonstrate 
satisfactory  level*  of 

1.5.1 

Knowing  what  learners 

job  performance  skills 

are  Intended  to 

2.5.2 

Facilitators  demonstrate 

achieve 

satisfactory  levels  of 

1.5.2 

Knowing  the  training 

instructional  skills 

content 

2.5.3 

Facilitators  demonstrate 

1.5.3 

Knowing  the  training 

satisfactory  abilities  to 

resources 

Implement  the  curriculum 

1.5.4 

Knowing  the  training 

for  a  given  delivery 

procedures  and  methods 

application 

1.5.5 

Knowing  the  training 
drcumstances 

1.6.1 

Adapting  the  curricu¬ 
lum  to  the  facilita- 

tor's  style  and 
strengths 

1.6.2 

Adapting  the  curricu¬ 
lum  to  the  specific 
learning  needs  of  a 
given  delivery  appli¬ 
cation 

1.6.3 

Adapting  the  curricu¬ 
lum  to  specific  con- 

•trainta  of  a  given 
delivery  application 

1.1. 1.1 

Identifying  scheduling 

2. 1.1.1 

Assess  prospect's  experl- 

option* 

ential  history  with 

1.1. 1.2 

Devising  schedule 

reaped  to  the  taaka 

1.11.3 

monitoring  mechanisms 

2. 1.1. 2 

Assets  prospect's  train- 

Devising  schedule 

Ing  with  reaped  to  the 

control  mechanisms 

2. 1.1. 3 

tasks 

Assess  degree  to  which 

1.1.2. 1 

Encouraging  maximum 

the  prospect  manifests 

1.1. 2. 2 

laarner  participation 

the  qualities  and  charac- 

Eliciting  aenae  of 

teristics  of  a  good  per- 

reaponaibillty  for 
learning  from  learner* 

former  of  the  tasks 

1.1. 2. 5 

Treating  learner*  with 

2. 1.2.1 

Teat  the  prospect'*  abil- 

1  1.2.4 

reaped 

Ity  to  satiafy  the  per- 

Encouraging  learner*' 

formancc  objective*  asao- 

freedom  of  expreaeion 

dated  with  each  task 

1.1. 2. 5 

Avoiding  disorder  and 

2. 1.2. 2 

Estimate  the  cost  of 

laarner  diataate 

training  that  would  be 
necessary  to  achieve 

1.1. 3.1 

Measuring  learner*' 

satisfaction  of  the 

1.1. 3. 2 

prograss 

performance  objectives 

Diagnosing  emerging 

2. 1.2. 1 

Formulate  decisions  to 

deficiencies  and  dlf- 

provide  or  not  provide 

Acuities  in  learning 

task  performance  train- 

1. l.S.J 

Simplifying  key 

ing  for  each  prospect 

operative  Information 

on  each  task 

1. 1.3.4 

1.1. 3. 5 

Tailoring  learning 
activities  to  speci¬ 
fic  experiences  of 
learners 

Tailoring  schedule  end 
sequence  of  learning 
to  learners'  capabili- 

2. 2. 1.1 

Assess  prospact's  experi¬ 
ential  history  with 
reaped  to  managing/ 
implementing  those  types 
of  learning  activities 

2. 2. 1.2 

Assess  prospect's  prior 

ties  and  needs 

training  relative  to 

1.1. 4.1 

Developing  skills  in 
the  use  of  relevant 

those  type*  of  learning 
activities 

Instruction aJ  aids 

performance 

PRODUCTS 


Fadlitators  knowledge¬ 
able  of  the  ingredient* 
of  the  learning 
objective* 

Role-model  performera 
of  the  ingredient* 
Fadlitator*  knowledge¬ 
able  of  the  activities 
that  can  eupport 
achievement  of  the 
learning  objective* 
Insurers  of  Jhe  ade¬ 
quacy  of  the  physical 
learning  environment 
Insurer*  of  the  avail¬ 
ability  of  needed  mate¬ 
rials  and  resource* 
Communicators  of  inter¬ 
est  and  enthusiasm  for 
the  learning 
Fadlltators  who  are 
respectful  of  learners 
Fadlitator*  who  are 
accepting  of  learners' 
needs  and  differences 
Fadlitator*  who  are 
supportive  and  encour¬ 
aging  of  learners' 
initiatives 

Facilitators  who  are 
accessible  to  learner* 

Elldtor*  of  learners' 
participation  in  the 
learning  planning 
procees 


PERFORMANCE 

POTENTIALITIES 


1.1. 4. 2 

Developing  a  kill*  In 
th«  appropriate  con¬ 
trol  of  thr  physical 
environment 

1.1. 4.1 

Developing  skills  In 
the  preparation  of 
retevanl  apecial 
facilities  and 
equipment 

1.1.  5.1 

Preparing  (camera 
to  learn 

1. 1.5.2 

Preaentlng  learning 
content  to  learners 

1.1. 5. 3 

Guiding  Isarners' 
application  of  learn¬ 
ing 

1.1.  S.l 

Testing  learners' 
achievement  of  learn¬ 
ing 

1. 1.6.1 

Assembling  learning- 
relevant  experiences 

1. 1.6.2 

Acquiring  and/or 
fabricating  resource 
materials 

1.2. 1.1 

Informing  learners  of 
the  learning  oh|»c- 
lives 

1.2. 1.2 

Relating  objectives 
to  learaera'  own 
needs 

1.2. 1.3 

Relating  objectives 
to  learners’  own 
experiences 

1.2. 2.1 

Gearing  presentations 
to  learners'  abili¬ 
ties  and  achievements 

1.2. 2. 2 

Selecting  meaningful 
Images,  patterns, 
etc. .  to  vivify 
content 

1.2. 2.1 

Utilising  multiple 
sense*  to  reinforce 
learning 

1.2. 2. 4 

Involving  learners 
actively  in  the 
preaentatton 

1.2.1. 1 

Exposing  each  step  In 
the  procedure 

1. 2.1.2 

Clarifying  the  pur¬ 
pose  for  each  step 

1.2.!.! 

Manifesting  perfor¬ 
mance  technique 

1.2. 1.4 

Involving  learners 
actively  In  the 
demonatration 

1.2. 4.1 

Selecting  and  creat¬ 
ing  opportunities  for 
practice 

1.2. 4. 2 

Involving  learners 
actively  In  practice 

1.2. 4.1 

i 

Sequencing  practice 
experiences  to 
enhance  learning 
development 

PERFORMANCE 

PROCESSES 


2. 2. 1.1 

Aaaeas  degree  to  which 
prospect  manifests  the 
qualities  and  character¬ 
istics  associated  adth  a 
good  manager /Implementor 
of  those  types  nf  learn¬ 
ing  activities 

2. 2. 2.1 

Test  the  prospect's  abil¬ 
ity  to  manage/implement 
the  learning  activity  in 
question 

2. 2. 2. 2 

Estimate  the  coat  of 
training  that  would  be 
needed  to  prepare  the 
prospect  to  manage/ 
Implement  each  activity 

2. 2. 2.1 

Formulate  decisions  to 
provide  or  not  provide 
facilitator  training  for 
each  prospect  on  each 
learning  activity 

2. 1.1.1 

Identify  apecific  perfor¬ 
mance  objectivea  for 
which  job  performance 
training  is  needed 

2.3. 1.2 

Analyte  the  objectives 
to  identify  the  learning 
events/activities  needed 

2. ’.1.1 

Select  implementing 
methoda/ media 

2. 1.1. 4 

Prepare /aasemble  de¬ 
tailed  prescriptions  of 
the  methods  and  media 

2. 3.7.1 

Identify  specific  perfor¬ 
mance  objectives  for 
which  facilitator  train¬ 
ing  is  needed 

2. 1.2. 2 

Analyte  the  objectives 
to  identify  the  learning 
eventg/activities  needed 

2. 1.2.1 

Select  Implementing 
methoda/media 

2. 1.2. 4 

Prepare/asaemble  de¬ 
tailed  prescriptions  of 
the  methoda  and  media 

2. 4. 1.1 

Identify  specific  per- 
formancs  objectivea  re¬ 
lating  to  familiarity 
with  the  curriculum  end 
Instructional  require¬ 
ments 

2.4. 1.2 

Analyse  the  objectives 
to  identify  the  learning 
events /activities  needed 

2. 4. 1.1 

Select  implementing 
methods/media 

2. 4. 1.4 

Prepare  /  assemble  de¬ 
tailed  prescriptions  of 
the  methods  and  media 

2. 4. 2.1 

Identify  specific  per¬ 
formance  objectives  re¬ 
lating  to  familiarity 
with  the  requiremente 
of  a  given  delivery 
application 

PERFORMANCE 

PRODUCTS 


1.1. 2. 2 

Creators  of  conditions 
to  onhance  desire  for 
learning 

1.1.2. 1 

Diagnosticians  of 
learning  needs 

1.1.2. 4 

Creators  of  experience 
to  enhance  learning 

1.1. 2. 3 

Counselors  of  learners 

1. 1.2.6 

Boosters  of  iearnera' 
aspirations  and  effort  a 

1. 1.2.7 

Managers  of  media  to 
aid  the  learning 
process 

1.1. 2.4 

Deliverers  of  informa¬ 
tion 

1. 1.2.4 

Coaches  of  learners 

1.1.2.10 

Ev«)uatort  of  learner*' 
progress 

1.2. 1.1 

Performance  objectives 
specifications  couched 

In  terms  relevant  to 
the  given  learners' 
needs 

1.2. 1.2 

Plana  for  learning 
activities /experiences 
that  are  In  accordance 
with  the  given  learn¬ 
ers'  needs 

1.2. 1.1 

Presentations,  appli¬ 
cations.  and  testa 

that  are  consistent 
with  the  given  learn¬ 
ers’  Intended  applica¬ 
tions  of  learning 

1.2.1. 4 

Complete  set*  of 
refinements  and  de¬ 
tail*  ncceaaarv  to 
Implement  the  curricu¬ 
lum  in  a  faahlon  con¬ 
sistent  with  Igarners’ 
needs 

1.2.2. 1 

Performance  prerequi¬ 
site  specifications 
couched  In  term*  rele¬ 
vant  to  the  given 
learner*'  experience* 

1.2. 2. 2 

Plana  for  Isarning 
activities  /experiences 
that  draw  upon  the 
given  learner*'  experi¬ 
ences 

1.2. 2.1 

Presentation*  and 
applications  that  make 
use  of  the  given 
learner*'  axperienca* 

Figure  8  (Continued) 


CHARACTERISTICS  (Cont’d) 


PERFORMANCE 

POTENTIALITIES 


PERFORMANCE 

PROCESSES 


PERFORMANCE 

PRODUCTS 


1.2. 4. 4 

Developing  end  main¬ 
taining  realism  in 
practice 

1.2. 4. 5 

Observing  and  tutor¬ 
ing  learner*  In 
practice 

1.2. 4. 6 

Helping  learner* 

»ense  progress  toward 
learning  objective* 
and  goal* 

1.2.5. 1 

Diagnoaing  specific 
needa  for  learning 

1.2. 5. 2 

Diagnosing  learner  a’ 
progreaa 

1.2.5.J 

Diagnosing  deficien¬ 
cies  in  the  learning 
environment 

1.3. 1.1 

Clarifying  the  value 
of  the  learning  to 
the  learnera 

1.3. 1.2 

Demonstrating  enthu- 

aiaam 

1.3. 1.3 

Encouraging  learnera' 
aaplrationa  and  Ini¬ 
tiatives 

1.3. 1.4 

Rewarding  learners' 
progress  and  partici¬ 
pation 

1.3.1. 5 

Demonstrating  pa¬ 
tience  and  compaasion 

1.3.2. 1 

Creating  aituatione 
and  learning  activi¬ 
ties  that  are  inher¬ 
ently  interesting 

1.3. 2. 2 

Conveying  a  eenae  of 
excitement 

1.3. 2. 3 

Dramatising  for  em- 
phaaia  and  rein¬ 
forcement 

1.3. 2. 4 

Sensing  and  respond¬ 
ing  to  learners' 
attention  and 
interest 

I. 3. 3.1 

Exploiting  learners' 
experiences  through 
discussions,  case 
studies,  role- 
playing.  etc. 

1.3. 3. 2 

Emphasizing  Immedi¬ 
acy  of  application 
of  teaming  to 
learner-relevant 
problem*/  needa 

1.3. 4.1 

Raising  end  malntein- 
ing  learners'  level 
of  aspiration  for 
achievement 

1.3. 4. 2 

Mediating  end  facili¬ 
tating  the  transfer 
frost  training  to 

spar  at  ton  a 

2. 4. 2. 2 

Analyze  the  objectives 
to  identify  the  learn¬ 
ing  eventa/activitiea 
needed 

2.4.2. 3 

Select  implementing 
methods /media 

2. 4. 2. 4 

Prepare/assemble  de¬ 
tailed  prescriptions  of 
the  methods  and  media 

2.4.3. 1 

Identify  apecific  per¬ 
formance  objectives 
relating  to  developing 
additional  details 

2. 4. 3. 2 

Analyse  the  objectives 

to  identify  the  learn¬ 
ing  events /activities 
needed 

2.4. 3. 3 

Select  implementing 
methods /media 

2. 4. 3. 4 

Prepare /aaaemble  de¬ 
tailed  prescriptions  of 
the  methods  and  madia 

2.  S.  1.1 

Implement  activities  to 
prepare  facilitator!  to 
achieve  the  needed  job 
performance  (kills 

2. 5. 1.2 

Implement  activities  to 
present/demonstrate  the 
akilli  to  facilitators 

2.5. 1.3 

Implement  activities  to 
insure  that  facilitators 
practice  the  skills 

2. 5. 1.4 

Implement  activities  to 
teat  the  facilitators’ 
achievement  of  the 
needed  job  performance 
skills 

2.5.2. 1 

Implement  activities  to 
prepare  facilitators  to 
achieve  the  needed 
learning  facilitation 
skills 

2. 5. 2. 2 

Implement  activities  to 
prssent/ demonstrate  the 
skills  to  the  facilita¬ 
tors 

2. 5. 2. 3 

Implement  activities  to 
insure  that  facilitators 
practice  the  skills 

2. 5. 2. 4 

Implement  activities  to 
test  facilitators' 
achievement  of  the 
needed  learning  facili¬ 
tation  skills 

2.5.3. 1 

Implement  activities  to 
prepare  facilitators  to 
achieve  the  needed  cur¬ 
riculum  Implementation 
abilities 

2.5.3. 2 

Implement  activities  to 
p  re  sen  I /demonstrate  the 
abilities  to  the  facili¬ 
tators 

CHARACTERISTICS  (Cont'd) 


I  * 


1.1. 4. 3 


1.1. 4. 4 


1.3. 5.1 


1.3. 5. 2 


1.1.5. 3 


1.3. 6. 2 


1.3.1. 3 


1.3.7. 1 


1.3. 7. 2 


1.1. 7. 3 


1.3. 7. 4 


PERFORMANCE 

POTENTIALITIES 


Senring  a*  a  mentor 
to  extend  and  control 
Warner*'  talent 
Prompting  and  cuing 
Warnera,  when  neces- 
aary 

Conveying  dlagnoaea 
of  learner*’  perfor¬ 
mance  to  learner* 
Poaltlvely  reinforc¬ 
ing  deairable  per¬ 
formance 

Conatructively  criti¬ 
cizing  deficient 
performance 

Aaaiating  learner*  to 
develop  and  apply 
aelf-evaluation  pro¬ 
cedure*  (uited  to 
learning  objective* 
Aaciating  learner*  to 
diagnose  their  own 
need*  for  learning 
Assisting  learners  to 
translate  diagnosed 
need*  into  specific 
objective*  and  learn¬ 
ing  activities 

Assisting  learners  to 
plan  and  organize 
their  learning 
activities 

Aaaiating  learners  to 
Implement  their  learn¬ 
ing  activities 
Assisting  learners  In 
dealing  with  problems 
affecting  their  learn¬ 
ing  progress/schieve- 
ment 

Interacting  with 
Warnera  to  bolster 
morale  and  raise  aspi¬ 
rations  as  learners' 
motivation  and  efforts 
diminish 

Knoaring  the  elements 
of  the  cognitive  (fac¬ 
tual)  aspects  of  the 
training 

Knowing  the  element* 
of  the  payehomotor 
(skill)  aspects  of  the 
training 

Knoaring  the  elements 
of  the  affective 
(attltudlnal)  aspects 
of  the  training 

Demonstrating  and 
articulating  the  cog¬ 
nitive  tasks  of  the 

l«*> 

Demonstrating  and 
articulating  the  psy- 
c  homo  tor  tasks  of  the 


PERFORMANCE 

PROCESSES 


2. 5. 1.1  Implement  activities  to 
insure  that  facilitators 
apply  the  curriculum 
Implementation  abilities 
2.5. 1.4  Implement  activities  to 
test  the  facilitators' 
curriculum  implementa¬ 
tion  abilities  and 
application  products 


PERFORMANCE 

PRODUCTS 


performance 

POTENTIALITIES 

PERFORMANCE 

PROCESSES 

PERFORMANCE 

PRODUCTS 

? 

s 

o 
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1.4. 2. 3  Demonstrating  and 

articulating  the  attl- 
tudaa  required  of  the 

Job 

1.4. 3.1  Relating  practical  ap¬ 
plication!  of  the  in¬ 
tended  learning  by 
role-model  performers 

1.4. 3. 2  Relating  beneficial 
learning  experience! 
ueed  by  previoua 
leamera 

1.4. 3. 3  Facilitating  inter¬ 
change  of  relevant 
experience!  of  current 
leamera 

1.5. 1.1  Underetanding  the  In¬ 

tended  goala  of  learn¬ 
ing 

1.5. 1.2  Underetanding  the  in¬ 

tended  performance 
objective! 

1.5. 1. 3  Underetanding  the  pre- 
requleite  abilitiea 
expected  of  leamera 

1.5. 2.1  Underetanding  the 
topic!  to  be  covered 

1.5. 2. 2  Underetanding  the 
aaeociatlonx  among 
toplce  and  performance 
objective! 

1.5. 2. 3  Undemanding  the  or¬ 

ganisation  of  the 
content'e  topical 
aequence 

1.5.3. 1  Undemanding  the  per- 

aonnel  reaourcea 
available  and  needed 

1.5. 3. 2  Undcratanding  the 

equipment  reaourcea 
available  and  needed 

1.5. 3. 3  Underatandlng  the 

material/media  re¬ 
source!  available  and 
needed 

1.5. 3. 4  Underetanding  the 

facilitiea  reaourcea 
available  and  needed 

1.5.4. 1  Underatandlng  the 

preparation  activities 
and  procedures  to  be 
used 

1.5. 4. 2  Understanding  the  pre¬ 
sentation  activities 

and  procedures  to  be 
used 

1.5. 4. 3  Underetanding  the  ap¬ 
plication  activities 
end  procedures  to  be 
uaed 

1.5. 4. 4  Understanding  the 
evaluation  activities 
end  procedures  to  be 

used 

Figure  8  (Continued) 
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CHARACTERISTICS  (Cont'd) 


PERFORMANCE 
POTENT!  AUT1ES 


PERFORMANCE 

PROCESSES 


PERFORMANCE 

PRODUCTS 


l.S.S.l 

Understanding  what 
conditions  ire  to  b« 
controlled  during 
learning  activities 

1.6.S.2 

Understanding  how  to 
control  thoae  condi¬ 
tion  a 

1.6. 1.1 

Capitalising  on  fa¬ 
cilitator  s'  own  expe¬ 
riences  that  are  rele¬ 
vant  to  the  intended 
learning 

1.6. 1.2 

Selecting  learning  ac¬ 
tivity  options  that 
are  compatible  with 
facilitators'  own  in¬ 
structional  skills 

1.6. 1.3 

Developing  procedural 
details  to  enhance  fa¬ 
cilitators'  delivery 
of  instruction 

1.6. 1.4 

Acquiring  and/or  re¬ 
fining  materials  to 
enhance  facilitators' 
delivery  of  instruc¬ 
tion 

1.6.2. 1 

Deleting  performance 
objectives  that  are 
not  relevant  to  the 
kerning  needs  of  a 
given  delivery  appli¬ 
cation 

1.6. 2. 2 

Defining  new  perfor¬ 
mance  objectives  rele¬ 
vant  to  the  learning 
needs  of  a  given 
delivery  application 

1.6.2.) 

Modifying  content  to 
reflect  changed  objec¬ 
tives 

1.6. 2. 4 

Modifying  resources  to 
reflect  changed  objec¬ 
tives 

1.6. 2. 5 

Modifying  procedures 
and  methods  to  reflect 
changed  objectives 

1.6. 2. 6 

Modifying  conditions 
to  reflect  changed 
objectives 

1. 6.1.1 

Modifying  resources  to 
reflect  specific  con¬ 
straints 

1.6.). 2 

Modifying  procadures 
snd  methods  to  reflect 
specific  constraints 

1.6.).) 

Modifying  conditions 
to  reflect  specific 
constraints 

Figrure 


(Continued) 
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under  Performance  Potentialitites,  each  population  category  is  identified  as  1.1, 

1.2,  1.3,  etc.  The  Functional  Purposes  Level  descriptions  is  assigned  a  two 
decimal  code,  i.e.,  1.1.1,  1.1.2,  1.1.3,  etc.  Finally,  the  Characteristics 
Level  of  system  description  is  assigned  a  three  decimal  code,  i.e.,  1.1. 1.1, 

1.1. 1.2,  1.1. 1.3,  etc.  The  decimal  numbering  of  the  population  categories 
simply  aids  in  organizing  the  population  categories  in  such  a  way  so  that  the 
analyst  can  easily  trace  the  subpopulation  categories  of  Objectives  Level, 
Functional  Purposes  Level  and  Characteristics  Level  populations. 

Subsequent  segments  of  this  report  illustrate  the  application  of  these  taxa 
to  produce  measures  and  design  specifications  for  representative  military 
training  systems.  To  prepare  for  those  sample  applications,  this  segment 
closes  with  brief  clarifications  of  the  design  and  measurement  implications  of 
one  taxonomic  subset  of  the  Design  subsystem.  This  clarification  also  will 
serve  to  elucidate  certain  of  the  terms  and  concepts  of  a  general  training 
system. 


Sample  Taxonomic  Subset 


Training  Subsystem: 
Performance  Category: 
Objectives-level  taxon: 


Design 

1.0  Performance  Potentialities 
"1.2  Establishing  Performance 
Objectives" 


Derivative  Functional  Purposes-level  taxa: 

"1.2.1  Stating  what  the  learners  will 
be  able  to  do  or  how  they  will  be 
expected  to  behave  after  completing 
training" 


"1.2.2  Providing  a  basis  for  objectively 
assessing  learner  performance" 

Derivative  Characteristics-level  taxa: 

"1.2. 1.1  Defining  the  performance  action"  "1.2. 2.1  Developing  pretest  items" 

"1.2. 1.2  Defining  the  performance  condition"  "1.2. 2. 2  Developing  posttest  items" 
"1.2. 1.3  Defining  the  performance  standard" 


One  of  the  many  fundamental  capabilities  that  the  Design  subsystem  is 
expected  to  provide  for  the  benefit  of  the  total  training  system  is  the  ability 
to  establish  performance  objectives  for  learning.  The  Command  subsystem 
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decrees  that  learning  will  take  place,  having  identified  a  "job"  for  which 
training  is  needed.  However,  Command  defines  the  "job"  only  in  broad  terms, 
e.g.,  "we  need  to  train  soldiers  to  man  the  Bradley  Infantry  Fighting 
Vehicle."  It  is  up  to  Design  to  translate  that  broad  statement  into  a 
behavioral  reference  framework  that  is  compatible  with  learning.  Given  that 
learning  is  sensory  activity  intended  to  affect  behavior,  one  of  the  Design 
subsystem's  first  responsibilities  is  to  establish  exactly  what  behaviors  are  to 
be  affected,  and  exactly  how  they  are  to  be  affected.  In  training  system 
terms,  that  is  called  "establishing  performance  objectives." 

Performance  objectives  are  simply  behaviors,  expected  of  the  learner  upon 
completion  of  training,  that  can  be  defined  and  observed  in  quantifiable  terms. 
Precise  definition  of  observable  behavior  is  absolutely  essential,  botn  to  ’ 
determine  the  training  content  and  methods  and  to  evaluate  training 
effectiveness.  A  training  designer  would  never  (or  should  never)  say 
something  like  "one  of  my  objectives  is  to  give  the  BIFV  gunner  trainees  an 
appreciation  of  the  kinds  of  targets  they  can  engage  with  the  weapons  on  this 
vehicle."  There  is  nothing  quantifiable  or  directly  observable  about 
"appreciation."  What  does  a  learner  have  to  do  to  acquire  an  "appreciation"? 
How  can  one  tell  whether  one  learner  has  achieved  more  "appreciation"  than 
another?  Of  course,  what  the  designer  really  means  is  that  the 
gunner-learners  should  come  to  know  what  targets  can  and  should  be  engaged 
by  each  type  of  weapon.  That  is  a  better  statement  of  the  training  purpose, 
but  it  still  isn't  expressed  in  behavioral  terms.  A  true  performance  objective 
expressing  what  the  designer  really  intends  might  be  stated  as  follows: 
"Given  a  list  of  the  types  of  targets  to  be  engaged,  the  learner  will  state  the 
appropriate  weapon /ammunition  to  be  used  to  engage  each  target."  The 
intended  behavior  is  a  series  of  audible  statements  of  weapon /ammunition  type, 
which  is  readily  quantifiable  as  to  the  accuracy  of  each  statement. 

In  the  absence  of  valid  performance  objectives,  no  one  really  knows  what 
the  intended  behavioral  outcome  of  the  training  is  supposed  to  be,  and  so  no 
one  can  properly  design  or  evaluate  the  sensory  activities  that  are  to  form  the 
training  experience.  Thus,  both  the  designer  and  the  evaluator  of  any 
training  system  must  concern  themselves  with  the  issue  of  the  Design 
subsystem's  ability  to  establish  performance  objectives.  At  the  top-most  level 
of  design  and  evaluation,  crucial  questions  such  as  the  following  must  be 
asked: 

0  Is  there  a  person  (or  persons)  assigned  to  the  Design  subsystem 
who  has  the  knowledge  and  skills  needed  to  establish  valid 
performance  objectives  for  training? 

°  Do  the  plans  submitted  by  Design  subsystem  for  Command's  review 
and  approval  call  for  the  establishment  of  performance  objectives? 

°  Do  those  plans  allocate  appropriate  manpower  and  other  resources  to 
the  establishment  of  performance  objectives? 
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°  Do  the  plans  provide  for  obtaining  the  input  information  necessary 
for  the  establishment  of  performance  objectives? 

Once  the  designer  and  evaluator  are  satisfied  (from  their  different 
perspectives)  that  the  basic  capability  of  "establishing  performance  objectives" 
has  been  provided  in  the  Design  subsystem,  their  attention  shifts  to 
ascertaining  how  and  whether  this  capability  can  satisfy  its  functional 
purposes.  The  Design  subsystem  is  expected  to  be  able  to  establish 
performance  objectives  for  two  reasons:  first,  so  that  it  will  be  possible  to 
state  what  the  learners  will  be  able  to  do  or  how  they  will  behave  subsequent 
to  training;  second,  so  that  it  will  be  possible  to  develop  an  objective  basis 
for  assessing  learner  performance  in  training.  At  this  level  of  detail,  the 
designer  and  evaluator  deal  with  such  issues  as: 

°  Does  the  basic  capability  for  establishing  performance  objectives  lend 
itself  to  specifying  learners'  terminal  behaviors? 

0  Do  the  plans  and  procedures  used  by  the  Design  subsystem  call  for 
specifying  learners'  terminal  behaviors? 

°  Is  the  basic  capability  for  establishing  performance  objectives 

actually  being  applied  to  prepare  for  learner  performance 
assessment? 

Finally,  the  designer  and  evaluator  shift  their  focus  to  the  technical 
details  necessary  to  apply  the  capability  of  establishing  performance  objectives 
to  achieve  the  two  functional  purposes.  Numerous  issues  come  to  their 
attention  at  this  stage,  including  the  following: 

°  Does  the  Design  subsystem's  capability  for  establishing  performance 
objectives  extend  to  the  specification  of  the  actions  that  the  learners 
are  to  perform?  Does  the  capability  for  specifying  the  actions  insure 
that  the  actions  are  observable  and  quantifiable?  Does  it  insure  that 
the  actions  specified  accurately  demonstrate  the  intended  behaviors? 

0  Does  the  capability  for  establishing  performance  objectives  extend  to 
the  specification  of  the  conditions  under  which  the  learners  are  to 
perform  the  actions?  Does  it  insure  that  the  conditions  specified  are 
relevant  to  actual  job  requirements? 

0  Does  the  capability  for  establishing  performance  objectives  extend  to 
the  specification  of  standards  of  acceptable  performance?  Does  it 
insure  that  the  standards  specified  are  appropriate  and  relevant  to 
actual  job  requirements? 

0  Does  the  Design  subsystem  possess  the  capability  of  developing 

means  of  testing  learners'  performance  prior  to  training?  Does  that 
capability  insure  that  the  test  items  reflect  the  specified  performance 
objectives?  Do  Design  subsystem  plans  call  for  establishing  and 
applying  such  pretest  items? 
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Does  the  Design  subsystem  possess  the  capability  of  developing 
means  of  testing  learners'  performance  after  or  at  completion  of 
training?  Does  that  capability  insure  that  the  test  items  reflect  the 
specified  performance  objectives?  Do  Design  subsystem  plans  call  for 
establishing  and  applying  such  posttest  items? 

Training  system  designers  and  training  system  evaluators  work  with  the 
very  same  system  performance  requirements.  The  designer's  genius  lies  within 
his  or  her  ability  to  select  the  approaches,  methods,  and  means  for  insuring 
that  the  rec  lirements  are  satisfied  by  the  system  being  built.  The  evaluator's 
genius  is  found  in  his  or  her  selection  of  the  approaches,  methods,  and  means 
of  determining  how  well  the  designer's  job  was  done.  After  they  have  both 
completed  their  work,  the  evaluator  is  required  to  "report  back"  his  or  her 
findings  so  that  the  designer  may  correct  any  deficiencies  that  may  have  been 
uncovered.  With  respect  to  the  Design  subsystem  performance  requirements 
surrounding  the  potentiality  of  "establishing  performance  objectives,"  the 
evaluator  could  submit  any  of  the  following  "reports"  to  the  designer: 

Possible  Report  No.  1: 

"Look,  you  never  saw  to  it  that  your  Design  subsystem  included  a 
capability  for  establishing  performance  objectives.  None  of  the 
people  that  you  assigned  to  the  subsystem  had  any  knowledge  of 
what  constitutes  a  valid,  useful  performance  objective,  and  none  of 
them  had  any  experience  in  conducting  the  kinds  of  front-end 
analyses  needed  to  identify  appropriate  objectives.  The  result  was 
that  the  so-called  job  and  task  analyses  that  they  performed  were 
badly  botched  and  are  just  about  useless,  and  their  so-called 
training  objectives  don't  deserve  the  name.  The  Design  subsystem 
has  failed  to  specify  what  the  learners  are  supposed  to  be  able  to 
do.  That  means  there  is  no  way  of  telling  whether  the  curriculum's 
content,  schedule,  sequence,  or  procedures  are  any  good  at  all." 

Possible  Report  No.  2: 

"Okay,  you  provided  the  basic  capability  for  establishing 
performance  objectives  when  you  assigned  John  Jones  and  Mary 
Smith  to  that  task.  They  have  solid  credentials  in  that  area; 
they're  good  analysts,  and  you  gave  them  sufficient  resources  to  get 
the  job  done.  They  produced  what  appears  to  be  a  very  good  set  of 
statements  concerning  the  learners'  intended  achievements:  the 
actions,  conditions,  and  performance  standards  are  well  described 
and  based  on  very  thorough  job  analyses.  The  problem  is,  you 
didn't  see  to  it  that  their  excellent  work  would  be  applied  to 
assessing  learner's  performance.  The  people  you  assigned  to  the 
task  of  developing  test  items  seem  never  to  have  even  considered  the 
performance  objectives,  and  certainly  never  consulted  John  or  Mary. 
At  least  half  of  the  stated  performance  actions  aren't  elicited  by  any 


of  the  test  items ,  either  pre-  or  post-.  Almost  all  of  the  actions 
that  are  elicited  are  called  for  under  conditions  that  aren't  even 
close  to  those  specified  in  the  performance  objectives.  The  net 
result  is  that  you  won't  be  able  to  tell  whether  your  learners  have 
actually  achieved  their  intended  learning." 

Possible  Report  No.  3: 

"John's  and  Mary's  assignment  to  the  Design  subsystem  provided  a 
solid  capability  for  establishing  performance  objectives,  and  you  were 
wise  to  task  them  also  with  the  assignment  of  preparing  pre-  and 
post-test  items  to  address  each  objective.  They're  good,  hard, 
skillful  workers,  and  they  produced  excellent  statements  of  the 
performance  actions  and  conditions  required  of  the  learners,  and 
some  very  well  constructed  test  items.  The  only  problem  is  that  you 
didn't  give  them  quite  the  support  they  needed.  Mary  tells  me  that 
she  and  John  just  didn't  have  enough  time  and  funds  to  do  a  proper 
analysis  of  the  job  performance  standards.  She  admits  that  some  of 
the  criteria  they  state  in  their  performance  objectives  are  largely 
wild  guesses.  I'd  suggest  that  you  scrape  up  some  more  funds  to 
allow  them  to  study  the  jobs  and  tasks  to  greater  depth,  so  that  we 
can  be  sure  that  the  performance  standards  reflect  what  really  is 
required  in  the  field.  Otherwise,  we  might  find  that  we  are  either 
undertraining  or  overtraining  our  learners,  and  we  might  be  giving 
them  tests  that  are  either  too  easy  or  too  hard." 

Possible  Report  No.  4: 

"You  did  a  fine  job  in  seeing  to  it  that  the  basic  capability  for 
establishing  performance  objectives  was  built  into  the  Design 
subsystem.  In  choosing  Mary  and  John  to  operate  that  subsystem, 
you  obtained  the  services  of  two  of  the  best  in  the  business.  They 
tell  me  that  you  saw  to  it  that  they  received  all  of  the  resources  and 
support  they  needed  to  do  the  job  the  right  way.  It's  clear  that 
your  system  had  everything  needed  to  produce  comprehensive,  well 
stated,  and  complete  performance  objectives,  and  to  reflect  the 
objectives  in  the  test  items  that  were  developed.  There  is  absolutely 
no  fault  to  find  with  this  element  of  your  system's  performance." 

These  are  not  all  of  the  possible  reports  that  an  evaluator  might  submit 
concerning  a  Design  subsystem's  potential  for  establishing  performance 
objectives,  but  they  do  span  the  full  range  of  possible  findings.  In  the  first 
report,  the  evaluator  finds  that  a  basic  required  potentiality  simply  is  absent 
from  the  system  being  studied:  this  particular  system  has  no  capability  for 
establishing  performance  objectives.  This  is  a  failure  detected  at  the 
objectives  level  of  design/measurement.  A  fundamental  element  of  the  "what" 
of  performance  is  unsatisfied.  Note,  too,  that  the  failure  is  diagnosed  as  a 
missing  potentiality.  The  evaluator  does  comment  on  the  fact  that  there  are 
process  failures  ("badly  botched  job  and  task  analyses")  and  product  failures 


("stated  performance  objectives  that  don't  deserve  the  name"),  but  it  is  clear 
that  the  evaluator  perceives  these  to  be  the  natural  results  of  the  more 
fundamental  potentiality  failure  ("none  of  the  people  had  any  knowledge  of  or 
experience  with  establishment  of  performance  objectives"'.  At  this  stage  the 
evaluator  is  focusing  on  a  taxon  of  potentiality.  In  h.  her  complete 
report,  detailed  comments  presumably  also  would  be  made  on  the  related 
process  and  product  taxa,  i.e.,  the  evaluator  would  document  exactly  how  the 
job  and  task  analysis  processes  were  "botched"  and  exactly  how  the  stated 
performance  objectives  products  were  deficient. 

In  Report  No.  2,  the  evaluator  finds  a  different  situation.  Here,  there  is 
evidence  that  the  essential  requirements  for  establishing  performance  objectives 
were  met.  Qualified  people  were  brought  to  the  assignment,  and  they  were 
given  the  tools  needed  to  do  the  job  of  producing  the  objectives.  Indeed,  the 
e  .  aluator  finds  that  they  did  produce  the  objectives.  However,  their 
capabilities  and  their  products  were  not  applied  to  the  task  of  providing  a 
basis  for  assessing  learner  performance.  This  is  a  failure  on  the  function  1 
purposes  level  of  design/measurement,  a  deficiency  in  the  "why"  of  this 
system's  performance.  A  key  potentiality  was  made  available  in  the  system, 
but  the  system  failed  to  apply  that  potentiality  properly. 

The  evaluator  finds  still  a  different  situation  in  the  third  report.  The 
system  has  the  basic  (objectives  level)  potential  for  establishing  performance 
objectives.  The  system  applies  that  potentiality  to  achieve  its  two  intended 
(functional)  purposes.  However,  detailed  analysis  discloses  that  the 
potentiality  breaks  down  at  the  characteristics  level  of  design /measurement, 
i.e,,  in  the  "how"  of  performance.  The  potentiality  for  establishing 
performance  objectives  is  deficient  in  one  characteristic,  namely,  the  ability  to 
specify  standards  of  performance.  The  evaluator  traces  this  to  an 
insufficiency  in  the  resources  provided  for  applying  the  capability,'  which 
probably  is  a  typical  reason  for  a  malfunction  on  the  characteristics  level. 

At  this  point  it  is  approrpiate  to  reflect  on  the  hierarchical  relationship 
that  binds  the  objectives,  functional  purposes,  and  characteristics  levels  of 
system  design/measurement.  It  is  clear  that  a  system  performance  failure  on 
the  objectives  level  automatically  implies  corresponding  failures  on  the 
functional  purposes  and  characteristics  levels.  Very  simply,  if  a  system  does 
not  possess  some  essential  potentiality,  or  does  not  carry  out  some  essential 
process,  or  does  not  deliver  some  essential  product,  there  is  no  way  that  the 
system  can  then  apply  that  potentiality,  process,  or  product  to  serve  its 
intended  purposes:  you  can't  use  what  you  don't  have.  Also,  the  absence  of 
the  potentiality,  process,  or  product  itself  certainly  implies  the  absence  of  all 
of  its  characteristics  that  are  required  for  any  intended  purpose.  However, 
the  mere  fact  that  the  system  satisfies  its  objectives-level  performance 
requirements  does  nut  guarantee  that  the  associated  functional  purposes  and 
characteristics  requirements  will  be  met.  The  system  might  have  all  of  its 
basic  potentialities,  carry  out  all  of  its  basic  processes,  and  produce  all  of  its 
basic  products,  but  fail  to  apply  them  for  the  required  purposes.  Further, 
those  basic  potentialities,  processes,  and  products  may  not  have  exactly  the 
right  characteristics  needed  to  satisfy  those  purposes. 


In  the  evaluator's  fourth  report,  a  happy  finding  is  documented.  The 
system  under  study  has  everything  necessary  to  insure  that  the  potentiality  of. 
establishing  performance  objectives  is  present  in  all  of  its  characteristic 
details,  and  will  be  applied  to  serve  its  intended  purposes.  The  evaluator 
commends  the  designer  on  this  particular  element  of  his  or  her  work. 

But  what  exactly  is  the  evaluator  saying  in  that  excerpt?  The  statement 
only  says  that  a  requirement  for  potential  performance  has  been  met.  The 
evaluator  has  worded  the  statement  in  a  way  that  carefully  avoids  saying 
anything  about  the  processes  carried  out  or  the  products  delivered. 
Potentiality  does  not  guarantee  achievement.  Any  person,  no  matter  how 
capable,  can  make  mistakes.  Any  machine  can  break  down.  In  the  context  of 
the  simple  example  we  have  been  using,  it  is  possible  that  John  Jones  and 
Mary  Smith  could  "botch  up"  a  job  analysis  in  spite  of  all  their  experience. 
The  system  designer  and  the  evaluator  cannot  be  satisfied  with  seeing  to  it 
that  the  system  has  the  potential  to  do  its  job.  They  also  have  to  see  to  it 
that  the  proper  processes  are  planned  and  implemented  and  that  the  proper 
products  are  specified  and  actually  delivered. 

This  brings  us  to  the  point  where  it  is  appropriate  to  reflect  on  the 
hierarchical  relationship  involving  the  three  aspects  of  system  performance, 
viz.,  potentialities,  processes,  and  products.  Their  relationship  is  similar  to 
that  involving  the  three  levels  of  system  description.  The  absence  of  some 
required  potentiality  will  give  rise  to  corresponding  breakdowns  in  the 
system's  processes  and  products.  If  a  basic  capability  is  missing,  the  process 
used  to  apply  that  capability  cannot  be  carried  out  successfully.  If  the 
process  cannot  be  carried  out,  the  fruits  of  the  process  (i.e.,  some  product) 
will  not  be  delivered  as  needed.  On  the  ether  hand,  the  potentiality  might  be 
there  but  the  processes  using  that  potentiality  might  nevertheless  go  awry.  A 
process  might  be  carried  out  without  fault,  but  the  resulting  product  might 
nevertheless  be  deficient.  The  expression  "garbage  in,  garbage  out"  familiar 
to  data  processors  refers  to  this  situation:  the  data  processing  might  be  fine 
but  the  output  products  might  be  worthless,  because  the  inputs  also  were  of 
no  value.  Thus,  just  as  each  level  of  system  description  must  be  designed 
and  evaluated  in  turn,  so  must  each  aspect  of  performance.  The  system 
designer  and  the  system  evaluator  should  always  proceed  from  general  to 
specific,  from  the  basic  to  the  detailed,  from  the  potential  to  the  actual,  to 
insure  that  all  appropriate  steps  are  taken  to  bring  about  a  system  that  does 
exactly  what  it  is  supposed  to  do.  Figure  9  illustrates  this  essential 
developmental  flow  within  the  general  systems  taxonomy  structure.  The  basic 
flow  for  either  design  or  measurement  applications  is  from  the  most  fundamental 
requirements  of  potentiality  to  the  most  detailed  requirements  of  product, 
always  insuring  that  all  intermediate  processes  and  purposes  receive  careful 
attention.  Sample  applications  of  this  concept  are  given  in  the  segments  that 
follow . 
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The  APM  for  Measurement  of  Training  System  Performance/ Effectiveness 


The  purpose  of  this  segment  is  to  illustrate  how  the  performance  taxa 
identified  for  any  particular  human-machine  system  can  be  applied  to  derive 
useful  measures  of  that  system's  performance/effectiveness.  The  approach 
taken  proceeds  through  the  following  steps: 

(1)  Define  precisely  the  system  of  interest. 

(In  this  example,  the  system  of  interest  will  be  one  subsystem 
of  one  portion  of  an  emerging  Army  training  system.) 

(2)  Define  precisely  the  measurement  application. 

(Here,  the  application  will  focus  on  one  aspect  of  the  training 
system's  planned  learning  activities.) 

(3)  Identify  the  particular  performance  taxa  that  are  relevant  to  that 
measurement  application. 

(Not  all  of  a  system's  performance  requirements  are  of  interest 
to  every  measurement  application.  This  example  will  illustrate 
the  procedures  to  be  used  to  determine  which  requirements  are 
relevant  and  which  are  not.) 

(4)  Establish  the  issues  and  implications  for  measurement  that  derive 
from  each  relevant  taxon. 

(The  derivative  issues  and  implications  clarify  exactly  what 
relevance  the  particular  performance  requirement  has  for  the 
measurement  application  at  hand.) 

(5)  losit  measures  that  will  address  each  issue  and  implications. 

(Here,  the  measures  will  be  workable  means  of  determining 
whether  the  training  system  satisfies  its  relevant  performance 
requirements  in  a  way  that  insures  the  adequacy  of  the 
particular  learning  activities  being  examined.) 

1.  The  System  of  Interest:  The  Design  Subsystem  of 
61 EV  Gunnery  Training 

The  U.S.  Army  Infantry  School  presently  is  developing  a  training 
system  that  will  be  employed  to  supply  qualified  operators  for  the  new  Bradley 
Infantry  Fighting  Vehicle  (BIFV).  This  new  Bradley  Infantry  Fighting  Vehicle 
Training  System  (BIFVTS)  will  prepare  soldiers  and  officers  for  assignments  as 
BIFV  squad  members  (skill  level  1  soldiers);  BIFV  drivers;  BIFV  gunners; 
BIFV  Commanders;  and  various  support  functions  on  various  levels  of  command 
(platoon,  company,  etc.).  Figure  10  depicts  the  network  of  other  systems 
that  interact  with  BIFVTS.  These  include: 

°  The  system  known  as  the  U.S.  Army  Infantry  School.  At  its 
present  stage  of  evolution,  BIFVTS  is  wholly  contained  within 
the  Infantry  School,  which  therefore  is  a  BIFVTS  suprasystem. 
As  will  be  shown  below,  the  school  presently  operates  the 
BIFVTS  Command  and  Design  subsystems. 


o 


Various  existing  and  possibly  emerging  infantry  training 
systems  other  than  BXFVTS.  Initially,  BIFVTS  will  design  and 
deliver  "add-on"  training  in  BIFV  operations,  which  training 
must  be  compatible  with  other  on-going  training  that  the  BIFV 
learner-operators  have  or  will  have  received.  At  a  later  stage 
of  evolution,  BIFVTS  will  be  integrated  with  other  infantry 
training  systems,  a  step  that  will  require  careful  coordination 
with  those  systems.  At  present,  those  systems  are  collateral  to 
BIFVTS. 

Various  U.S.  Army  infantry  units.  BIFVTS  learners,  facilitators 
and  other  personnel  will  be  drawn  from  infantry  units.  Those 
units  thus  are  human-machine  systems  collateral  to  BIFVTS. 

The  Fort  Benning  Logistic  Support  System.  This  entity  works 
closely  and  often  with  the  Infantry  School,  although  it  is 
functionally  distinct  from  the  school.  The  Fort  Benning 
Logistic  Support  System  will  operate  a  significant  portion  of  the 
BIFVTS  Logistics  subsystem,  especially  during  the  early  stages 
of  BIFVTS  operation.  In  the  future,  as  greater  shares  of 
BIFVTS  operation  move  "into  the  field,"  the  infantry  units  will 
take  over  increasingly  more  of  the  Logistics  functions  in 
support  of  training. 

The  BIFV  itself,  together  with  its  full  complement  of  personnel 
and  equipment.  The  BIFV  clearly  is  an  essential  collateral 
system  to  BIFVTS. 

The  Bradley  Cavalry  Fighting  Vehicle  Training  System 
(BCFVTS).  This  is  a  fraternal  twin  of  BIFVTS,  and  many  of 
its  operations  and  performance  requirements  will  be  identical  to 
those  of  BIFVTS.  Pooling  of  resources  and  sharing  of 
information  and  products  between  these  two  systems  is  highly 
desirable  if  not  absolutely  essential.  (Parenthetically,  BCFVTS 
is  itself  imbedded  in  an  interactive  network  of  systems  that 
exactly  parallels  the  BIFVTS  network.  A  portion  of  the 
BCFVTS  network  is  also  shown  in  Figure  10.) 

Since  BIFVTS  is  a  training  system,  it  must  (according  to  one  of  the 
fundamental  axioms)  incorporate  the  six  general  training  systems  within  itself. 
How  are  those  subsystems  structured  and  operated  in  BIFVTS? 

The  BIFVTS  Command  subsystem  is  operated  and  equipped  by  the 
BIFV  Task  Force,  an  ad  hoc  subsystem  of  the  U.S.  Army  Infantry  School. 

The  BIFV  Task  Force  has  been  created  and  staffed  to  exercise  overall 
responsibility  for  the  design,  development,  testing,  and  final  production  and 
dissemination  of  BIFV  training.  The  specific  immediate  responsibilities  of  the 
Task  Force  are  to: 


o 


Identify  the  specific  skills  and  knowledge  required  of  BIFV 
operators. 

Develop  BIFV  training  strategies. 

Establish  new  equipment  training  (NET)  for  the  BIFV. 

Administer,  monitor,  and  evaluate  all  BIFV  training 
development,  testing,  and  production. 

The  reader  will  recognize  that  several  of  the  above-listed 
responsibilities  are  actually  training  design  functions.  Indeed,  the  BIFVTS 
Design  subsystem  also  is  operated  (at  present)  solely  by  members  of  the  BIFV 
Task  Eorce.  Their  current  design  work  is  oriented  toward  the  production  of 
five  distinct  curricula: 

0  An  "add-on"  course  for  Skill  Level  1  BIFV  Soldiers  (MOS 
1 1 M 1 0 ) . 

°  A  Basic  Gunnery  Course  (for  Skill  Level  2  Soldiers  [MOS 
11M20]  and  some  Skill  Level  1  Soldiers). 

°  A  BIFV  Commander's  Course  (for  Skill  Levels  3-5  [11M30, 

11M40,  11M50]  and  for  officer  grades  01-05). 

°  A  transitional  new  equipment  training  (NET)  course  covering 
the  above-listed  skill  levels. 

°  A  Master  Gunner's  Course. 

The  Master  Gunner's  curriculum  principally  is  intended  as  an  input 
to  the  BIFVTS  Enabling  subsystem.  One  E7  or  E8  from  the  S3  section  of  each 
BIFV  mechanized  infantry  battalion  will  receive  Master  Gunnery  training,  as 
will  one  E6  from  each  BIFV  company.  The  Master  Gunner's  duties  include  the 
administration,  supervision  and  delivery  of  Basic  Gunnery  training  within  his 
or  her  unit.  The  Master  Gunner,  in  effect,  will  be  a  field 
facilitator /administrator  within  BIFVTS.  Other  current  or  planned  operations 
within  the  BIFVTS  Enabling  subsystem  include  the  organization  and  deployment 
of  a  team  of  facilitators  to  support  new  equipment  training  on  a  transitional 
basis  as  the  BIFV  is  introduced  into  operational  units;  familiarizing  selected 
mechanized  infantry  instructor/facilitators  with  the  emerging  BIFV  curricula; 
and  delivering  Mastery  Gunnery  training. 

It  is  not  presently  envisioned  that  the  BIFVTS  Enabling  subsystem 
will  provide  formal  instructional  skills  training  to  all  selected  facilitators. 
Instead,  persons  who  are  selected  to  serve  as  BIFVTS  facilitators  will  be 
recruited  from  among  those  who  already  possess  instructional  delivery  and 
guidance  skills.  Note,  however,  that  this  approach  in  no  way  changes  the 
performance  requirements  faced  by  the  Enabling  subsystem.  That  subsystem 
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still  must  have  the  potentiality  for  insuring  that  adequate  instructional  skills 
resources  are  provided  for  BIFVTS  Delivery  applications.  It  still  must  carry 
out  the  processes  of  assessing  the  candidate  facilitators'  instructional  skills 
and  of  diagnosing  their  needs  for  improvement  and  remedial  training.  It  still 
and  always  has  the  responsibility  of  producing  facilitators  who  are  qualified  in 
every  respect  for  their  assignments.  The  decision  to  recruit  experienced 
instructors  rather  than  to  provide  basic  instructional  training  to  novices  is  a 
system  design  choice  that  may  or  may  not  be  optimum  in  a  particular  case, 
depending  on  the  exact  circumstances.  The  design  choices  taken  are  simply 
the  variable  means  to  the  immutable  ends  represented  by  the  performance 
requirements,  or  taxa.  The  system  evaluator  is  far  less  interested  in  the 
approaches  the  designer  took  and  in  why  those  approaches  were  taken  than  he 
or  she  is  in  determining  whether  the  system — as  designed--does  the  job  it  is 
supposed  to  do.  It  may  well  be  (and  probably  is)  an  excellent  decision  of  the 
BIFV  Task  Force  to  recruit  experienced  instructors  to  serve  as  BIFV  training 
facilitators.  But  that  decision,  for  good  or  ill,  does  not  lessen  the  BIFVTS 
Enabling  subsystem's  responsibility  for  providing  fully  qualified  facilitators. 

The  BIFVTS  Emplacement  subsystem  is  responsible  for  the  design 
and  selection /fabrication  of  all  audiovisuals  and  other  learning  aids  necessary 
to  support  implementation  of  the  various  curricula,  and  for  the  acquisition  or 
construction  and  preparation  of  all  sites  at  which  learning  activities  will  take 
place,  whether  in  a  classroom  or  field  setting.  One  important  element  of  this 
responsibility  will  be  the  design  and  construction  of  gunnery  practice  ranges. 
The  range  requirements  presently  envisioned  include: 

0  Firing  port  weapons  range 

°  BIFV  Basic  Gunners  range 

°  BIFV  Squad  subcaliber  range 

°  BII'V  Squad  qualification  range 

°  Platoon  qualification  attack  range 

°  Platoon  qualification  defense  range 

Note  that  all  of  the  BIFVTS  Emplacement  subsystem's  performance 
requirements  derive  from  the  need  to  insure  that  all  facilities  and  materials 
necessary  to  support  delivery  of  BIFV  training  are  available  and  operable  at 
the  times  and  places  needed.  The  Emplacement  subsystem  designers  might 
determine  that  the  best  way  to  insure  the  availability  of  ranges  where  and 
when  needed  is  to  construct  six  separate  ranges,  one  for  each  gunnery 
requirement.  Conversely,  they  might  analyze  the  proposed  Delivery  schedules 
and  conclude  that,  two  physical  ranges  will  suffice,  time-sharing  the  six 
requirements  between  them.  The  former  approach  might  satisfy  the 
subsystem's  product  performance  requirements,  but  if  it  turns  out  that  the 
ranges  are  significantly  underutilized,  the  evaluator  may  conclude  that  certain 
analytic  process  performance  requirements  were  not  met,  resulting  in  an 
expensive,  wasteful  Emplacement  subsystem.  Similarly,  the  two-ranges 
approach  might  turn  out  to  have  been  a  brilliant  design  choice,  but  not  if 
severe  scheduling  conflicts  arise  because  the  two  ranges  cannot  handle  the 
training  load. 
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The  BIFVTS  Logistics  subsystem  presently  is  operated  solely  as  a 
component  of  the  Fort  fienning  logistic  support  system.  In  time,  increasingly 
greater  shares  of  the  operation  of  that  subsystem  will  be  exercised  by  various 
mechanized  infantry  units  in  the  field. 

The  BIFVTS  Delivery  subsystem  will  allow  and  insure  that  learning 
takes  place  in  the  contexts  of  the  various  curricula  produced  by  the  Design 
subsystem.  Soldiers  and  officers  will  carry  out  prescribed  sensory  activities, 
guided  and  assisted  by  facilitators,  and  will  achieve  the  specified  performance 
objectives  applicable  to  their  BIFV  operational  assignments. 

BIFVTS  th-  s  is  a  well  structured  training  system,  possessing 
functionally  oriented  subsystems  compatible  with  the  general  training  model  and 
pursuing  a  broad  range  of  administrative,  developmental,  and  delivery  goals  to 
support  the  introduction  of  the  new  BIFV  into  the  mechanized  infantry.  It 
does  not  detract  at  all  from  that  image  to  note  that  BIFVTS  can  also  be  viewed 
as  a  loose  consortium  of  smaller  scale,  relatively  independent  training  systems 
that  pursue  more  narrowly  defined  goals.  These  smaller  systems  are  organized 
around  the  five  BIFVTS  curricula  now  in  development.  These  systems  may  be 
denoted  as: 

°  The  BIFV  NET  System 

°  The  Basic  BIFV  Soldier  Training  System 

°  The  Basic  BIFV  Gunnery  Training  System 

°  The  Master  BIFV  Gunnery  Training  System 
°  The  BIFV  Commander  Training  System 

Each  of  these  has  its  own  Command,  Design,  Enabling,  Emplacement,  Logistics, 
and  Delivery  subsystems.  Their  development  is  proceeding  more  or  less  in 
tandem,  and  there  is  significant  overlap  in  their  objectives,  content,  and 
learning  activities,  as  well  as  in  their  administration.  But  they  are 
independent  in  the  sense  that  any  one  of  them  could  be  developed  and 
operated  in  the  absence  of  the  others.  (Admittedly,  there  wouldn't  be  much 
point  in  designing,  developing,  and  implementing  Master  Gunnery  training  if 
there  were  to  be  no  Basic  Gunnery  training,  but  it  still  could  be  done.) 

In  order  to  reduce  the  scope  of  this  trial  application  to  a  practical 
level,  the  project  staff  elected  to  narrow  their  focus  to  the  Basic  BIFV 
Gunnery  Training  System.  Like  all  systems,  this  one  operates  in  the  context 
of  an  interactive  system  network  of  suprasy stems,  collateral  systems,  and 
subsystems.  The  network  is  shown  in  Figure  11.  The  Basic  Gunnery  system 
is  subservient  to  such  entities  as  the  U.S.  Army  Infantry  School,  the  Fort 
Benning  Logistic  Support  System  and,  of  course,  the  total  BIFVTS.  It 
interacts  collaterally  with  the  other  four  independent  BIFV  training  systems 
and  with  portions  of  U.S.  Army  Infantry  units,  namely,  those  portions  from 
which  the  E4's  and  E5's  will  be  recruited  as  candidate  gunners.  It  also 
interacts  collaterally  with  the  BIFV  Carrier  Team  subsystem.  That  is  the 
portion  of  the  total  BIFV  system  for  which  the  gunner-learners  are  to  become 
qualified  as  operators.  And  finally,  the  Basic  BIFV  Gunnery  Training  System 


has  a  fraternal  (collateral)  twinship  with  the  Basic  BCFV  Gunnery  Training 
System.  The  subsystems  of  Basic  BIFV  Gunnery  Training  are,  of  course,  the 
six  familiar  subsystems  of  Command,  Enabling,  Design,  Emplacement,  Logistics, 
and  Delivery. 

Even  this  narrowed  focus  remains  too  broad  for  a  trial  measurement 
application.  Thus  far  in  the  present  research,  performance  taxa  have  been 
identified  only  for  two  of  the  six  training  subsystems.  The  tools  are  not  yet 
available  to  permit  assessment  of  a  total  training  system's  performance,  not 
even  one  concerned  with  such  a  specific  issue  as  BIFV  Gunnery  training.  It 
is  necessary  to  narrow  the  focus  further,  to  select  measurement  issues  lodged 
strictly  within  one  of  the  two  subsystems  for  which  taxa  now  are  known. 

We  choose  to  address  the  Design  Subsystem  of  BIFV  Basic  Gunnery 
Training,  rather  than  the  Enabling  Subsystem,  because  of  the  following 
considerations : 

Design  operations  are  at  a  further  stage  of  development  than 
are  Enabling  operations.  Any  inputs  that  this  research  can 
provide  for  assessing  Design  performance  would  be  more  timely 
*  for  the  BIFV  Task  Force  than  would  inputs  to  assessment  of 
Enabling  performance. 

Many  of  the  performance  requirements  of  the  Enabling 
subsystem  relate  to  the  facilitators'  interaction  with  the 
products  of  the  Design  subsystem.  Thus,  it  is  logical  to 
establish  first  that  Design  has  achieved  its  performance 
requirements  before  attending  to  Enabling  subsystem 
performance. 

As  a  collateral  to  all  of  the  other  training  subsystems.  Design 
performance  has  a  marked  effect  on  the  work  that  the  other 
subsystems  can  accomplish.  Any  strides  that  can  be  made  now 
toward  measurement  of  the  Gunnery  Design  subsystem  will 
provide  a  firmer  base  for  future  applications  of  the  APM  to 
BIFVTS  and  other  training  systems. 

Figure  12  depicts  the  network  of  systems  that  interact  most  closely 
with  the  Design  subsystem  of  Basic  BIFV  Gunnery  Training.  As  in  any 
training  system.  Gunnery  Training  Design  is  subservient  to  the  Command 
subsystem,  and  collateral  to  the  Enabling,  Emplacement,  Logistics,  and 
Delivery  subsystems.  Key  sub-subsystems  of  both  Emplacement  and  Delivery 
have  been  included  in  Figure  12  to  emphasize  the  special  gunnery  instruction 
and  practice  requirements  that  must  be  reflected  in  Design's  products. 
Gunnery  Training  Design  is  also  collateral  to  the  Design  subsystems  of  the 
other  BIFV  Training  "packages."  This  is  because  all  of  those  packages  share 
some  performance  objectives,  learning  activities,  etc.,  with  gunnery  training. 
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Figure  12.  Sample  Hierarchical  Structure  Centered  on 
a  Subsystem  of  a  Training  System  (Cont'd) 
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The  sub-subsystems  of  Gunnery  Training  Design  correspond  to  the 
thus  far  identified  segments  of  the  gunnery  curriculum.  These  represent  the 
sequential  stages  of  learning  through  which  BIFV  gunner-learners  must  pass  to 
become  fully  qualified.  The  initial  stages  stress  individual  training;  the 
latter,  team  training. 

2.  The  Measurement  Application:  Specification  of 

Gunner-Learner  Testing  "" 

Having  selected  the  Design  subsystem  of  Basic  BIFV  Gunnery 
Training  as  the  system  of  interest  for  this  trial,  it  remains  to  select  an  issue 
of  focus  for  the  application.  It  would  be  possible,  if  sufficient  resources  were 
available,  to  examine  the  global  performance  of  the  Design  subsystem,  i.e., 
simply  to  ask,  "How  well  does  the  Gunnery  Training  Design  meet  all  of  its 
performance  requirements?"  Within  the  scope  of  this  present  study,  and  given 
the  current  state  of  APM  application  procedures,  development  of  a 
comprehensive  and  complete  measures  set  would  be  unreasonably  consumptive 
of  project  time  and  funds.  A  more  restricted  measurement  application  seems  in 
order,  i.e.,  an  assessment  of  how  well  Design  meets  some  limited  aspect  of  its 
performance  requirements.  Of  course,  it  is  highly  desirable  to  insure  that  the 
aspect  chosen  for  study  is  of  interest  and  value  to  the  BIFV  Task  Force. 

What,  then,  are  some  typical  aspects  of  Gunnery  Training  Design 
performance?  What  are  the  issues  that  might  be  of  interest  to  system 
designers  and  evaluators? 

Recall  that  the  fundamental  function  of  any  training  system  is 
learning .  Everything  that  goes  on  anywhere  in  any  training  system,  or  in  any 
of  its  subsystems,  is  supposed  to  contribute  in  some  way  to  learning. 
Anything  within  the  system  that  detracts  from  or  degrades  learning  is  a 
system  defect.  At  rock  bottom,  all  assessments  of  training  system  or 
subsystem  performance  must  supply  some  portion  of  the  answer  to  this 
fundamental  question:  Did  the  learners  learn  what  they  were  supposed  to 
learn? 


What  does  Design  contribute  to  learning?  What  light  can  an 
examination  of  Design  performance  shed  on  the  quality  and  quantity  of  learning 
that  the  system  delivers? 

Design's  basic  job  is  to  provide  training  specifications.  It  specifies 
what  is  to  be  learned,  i.e.,  the  goals  am3  (especially)  the  performance 
objectives  that  learners  are  supposed  to  achieve.  It  specifies  the  context  of 
learning ,  i.e.,  the  content  of  knowledge,  abilities,  and  attitudes  required  of 
the  learning  and  the  informational  material  that  manifests  the  content.  It 
specifies  the  learning  activities,  i.e.,  the  sensory  experiences  that  will  affect 
the  learners'  behavior  in  the  intended  manner.  The  quality  and  quantity  of 
learning  will  suffer  to  the  extent  to  which  Design  badly  specifies  what  is  to  be 
learned  (e.g.,  produces  poorly  or  erroneously  stated  performance  objectives); 
or  to  the  extent  to  which  Design  badly  specifies  the  learning  context  (e.g., 
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by  including  topical  material  that  has  no  bearing  on  the  performance 
objectives,  thereby  confusing  the  learners);  or  to  the  extent  to  which  Design 
badly  specifies  the  learning  activities  (e.g.,  devises  activities  that  produce 
unexpected,  deleterious  effects  on  learners'  behavior).  Any  issue  that 
pertains  to  some  component  of  Design's  specification  of  the  learning  objectives, 
context,  or  activities  is  one  very  worth  exploring. 

The  issue  selected  for  this  trial  application  is  Gunnery  Training 
Design  performance  in  specifying  gunner-learner  testing.  That  is  a  sub-issue 
of  its  performance  in  specifying  learning  activities.  The  other  sub-issues  of 
that  heading  are:  specifying  gunner-learner  preparation  activities;  specifying 
activities  for  making  presentations  (or  demonstrations)  to  gunner-learners;  and 
specifying  gunner-learner  application  (or  practice)  activities.  Virtually  all 
learning  theorists*  agree  that  preparation,  presentation,  application,  and 
♦See  Bibliography  for  representative  documents. 

evaluation  (or  testing)  constitute  the  four  classes  of  learning  activities. 

The  issue  of  how  well  a  Design  subsystem  specifies  learner  testing 
procedures  is  of  special  importance  in  any  training  system.  Certainly,  that 
issue  will  affect  the  appropriateness  of  decisions  made  by  facilitators  and 
learners  themselves  concerning  whether  particular  learners  need  more  training, 
or  any  training  at  all,  to  achieve  particular  performance  objectives.  Even 
rno^e  significantly,  it  will  affect  the  appropriateness  of  the  system's 
certification  of  learners  as  c.  ipetent  job  performers,  thus  affecting  the  entire 
training  system's  reputation.  Most  basically,  if  deficient  learner  testing 
specifications  are  produced  by  the  Design  subsystem,  the  entire  training 
system  will  be  at  a  loss  as  to  assessing  its  most  fundamental  performance 
requirement:  it  will  not  really  know  whether  the  learners  have  learned. 

Thus,  no  training  system  can  afford  to  allow  its  Design  subsystem  t<?  produce 
poor  measures  of  learners'  achievements.  The  Basic  BIFV  Gunnery  Training 
System  is  no  exception. 

Before  moving  on  to  the  next  step  in  this  trial  application,  in  which 
particular  performance  taxa  associated  with  learner  testing  specification  will  be 
identified,  it  is  worthwhile  to  set  tne  stage  by  informally  reviewing  some  of  the 
requirements  and  implications  of  learner  testing  in  any  training  application. 
The  following,  at  least,  seem  almost  self-evident: 

Testing  must  be  objectives-referenced 

The  purpose  of  testing  is  to  evaluate  learners'  progress  toward 
or  achievement  of  the  performance  objectives,  i.e.,  that  which 
is  to  be  learned.  No  performance  objective  should  go  untested. 
No  test  that  does  not  relate  to  some  performance  objective 
should  be  included. 
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Testing  must  be  criteria-referenced 

It  is  not  enough  that  a  test  simply  requires  the  learner  to 
exercise  the  behavior  called  for  in  a  performance  objective. 
The  test  must  disclose  whether  the  behavior  is  performed  "well 
enough"  to  satisfy  learning  needs. 

Some  kind  of  pre-training  testing  is  necessary 

All  training  assumes  some  prerequisite  behavior.  If  nothing 
else,  it  is  always  assumed  that  the  learner  is  able  to  carry  out 
the  sensory  activities  called  for  in  the  training.  Many  training 
applications  admit  the  possibility  that  some  learners  may  already 
have  achieved  certain  of  the  specified  performance  objectives. 
There  is  no  point  in  commencing  training  unless  the  learner 
first  is  tested  to  insure  that  he  or  she  possesses  the  assumed 
prerequisites.  A  very  informal  test  might  suffice  for  that 
purpose,  perhaps  only  a  records  check  or  even  a  cursory 
visual  examination  of  the  learner,  but  some  testing  for 
prerequisites  surely  is  needed.  When  there  is  reason  to  believe 
that  some  learners  may  already  have  achieved  the  learning 
objectives,  pre-testing  of  those  objectives  also  will  be 
necessary. 

Some  testing  for  feedback  is  necessary 

Testing  is  after  all  a  learning  activity.  Learners  need  to  know 
how  well  they  are  doing  in  order  to  progress  and  persevere. 
Any  training  system  should  provide  at  least  some  testing  for 
the  purpose  of  feeding  back  progress  and  diagnosis  information 
to  the  learners. 

Testing  must  diagnose  learning  deficiencies 

It  is  not  sufficient  that  tests  disclose  that  a  learner  cannot 
perform  the  intended  behavior.  In  order  to  provide  meaningful 
feedback  to  the  learner  and  to  facilitate  subsequent  improvement 
of  the  curriculum  and  the  delivery,  tests  that  diagnose  the 
specific  ingredients  of  substandard  learner  performance  must  be 
included  in  the  specifications.  Only  then  can  the  learner's 
performance  be  corrected  and  brought  up  to  standard. 

Testing  must  be  reflective  of  on-the-job  performance 

The  tests  specified  must  require  the  learners  to  manifest  the 
job-relevant  behaviors  under  job-relevant  conditions.  The 
testing  conditions  should  resemble  job  conditions  as  closely  as 
possible,  since  the  test  outcome  otherwise  may  not  be  a  valid 
predictor  of  the  learner's  job  performance. 
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Some  kind  of  qualification  testing  is  necessary 

No  learner  should  be  "certified"  as  having  successfully 
completed  training  unless  he  or  she  demonstrates,  through 
testing,  that  the  performance  objectives  have  been  achieved. 
Depending  on  the  circumstances  of  the  particular  training 
experience,  the  qualification  testing  might  be  formal  or 
informal,  but  some  testing  surely  is  needed. 

The  preceding  list  is  an  "off-the-top-of-the-head"  compilation  of 
factors  that  probably  would  occur  to  any  evaluator  interested  in  measuring  the 
adequacy  of  a  Design  subsystem's  learner  testing  specifications.  What  follows 
is  an  attempt  to  impose  analytic  rigor  upon  the  measurement  process. 

3. 


Two  hundred  and  eighty-one  taxa  of  Design  Subsystem  performance 
requirements  are  listed  in  Figure  7,  above.  These  are  distributed  among  the 
three  aspects  of  performance  (potentialities,  processes,  and  products)  and 
across  the  three  levels  of  system  description  (objectives,  functional  purposes, 
and  characteristics).  Some  of  the  281  taxa  are  relevant  to  evaluation  of  a 
Design  Subsystem's  performance  in  specifying  learner  testing.  Others  are  not. 
At  the  present  stage  of  development  of  the  APM,  well  defined  procedures  for 
determining  which  taxa  do  or  do  not  bear  on  a  given  measurement  application 
are  yet  to  be  established.  Indeed,  one  purpose  of  this  trial  application  is  to 
add  to  the  base  of  experience  from  which  such  procedures  may  be  derived. 
Currently,  assessment  of  taxa  relevance  is  a  "judgment  call"  by  an  analyst  or 
team  of  analysts  working  together.  The  thought  process  can  be  described  as 
follows:  "Here  we  have  some  aspect  of  this  system's  performance,  and  over 

here  we  have  a  performance  taxon  for  the  system.  That  taxon  represents  a 
particular  requirement  that  the  system  is  supposed  to  satisfy.  Suppose  the 
system  does  a  bad  job  in  trying  to  satisfy  that  requirement.  Might  that  have 
any  bearing  on  the  particular  aspect  that  we  are  studying?"  If  the  analyst 
concludes  that  the  answer  is  "yes,"  the  taxon  in  question  would  be  included 
among  the  subset  from  which  measurement  issues  and  implications,  and 
ultimately  measures,  will  be  derived.  If  the  answer  is  "no,"  the  particular 
taxon  will  supply  no  input  to  the  measurement  application  at  hand. 
Identification  of  relevant  taxa,  then,  is  a  step-by-step  march  through  an 
entire  system  taxonomy,  during  which  each  taxon  is  picked  up  in  turn, 
examined  from  various  angles,  and  finally  accepted  or  rejected  in  accordance 
with  whether  or  not  it  appears  to  the  analyst  to  have  something  of  interest  to 
say.  This  process  can  be  streamlined  somewhat,  simply  by  examining  only  the 
characteristics-level  taxa.  If  a  taxon  on  the  characteristics  level  is  found  to 
be  relevant  to  the  particular  measurement  application,  then  the  functional 
purposes-level  taxon  from  which  it  derives  also  must  be  relevant.  A 
characteristics-level  taxon,  after  all,  is  simply  a  detailed  performance 
requirement  that  the  system  must  satisfy  if  it  is  to  achieve  the  related 


D  esign  Sub  system  Performance  Taxa  Relevant  to 
Measurement  of  the  Adequacy  of  Gunner-learner 
Testing  Specifications 
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1  ° 


functional  purpose.  Thus,  the  functional  purpose  must  be  subsumed  under 
the  aspect  of  performance  being  studied.  Similarly,  if  a  functional 
purposes-level  taxon  is  relevant  to  the  measurement  application,  so  must  be 
the  objectives-level  taxon  from  which  it  derives.  The  functional  purpose, 
after  all,  is  only  a  required  application  of  the  basic  potentiality,  process,  or 
product  defined  on  the  objectives  level.  For  example,  if  the  analyst  in  using 
Figure  7  were  to  conclude  that  taxon  number  2. 1.3.1  (a  characteristics-level 
process  requirement)  is  of  significance  to  the  measurement  application,  he  or 
she  would  automatically  know  that  taxa  2.1.3  and  2.1  (its  functional 
purposes-level  and  objectives-level  antecedents)  also  are  relevant.  These 
inter-relationships  are  described  in  more  detail  in  the  Year  1  Final  Report 
(Bloom,  et.  al. ,  1981). 

Appendix  A  of  this  report  documents  the  examination  of  every 
characteristics-level  taxon  of  potentiality  of  the  general  Training  Design 
subsystem  for  possible  relevance  to  assessment  of  the  subsystem's  performance 
in  specifying  learner  testing.  Rationale  for  concluding  whether  an  individual 
taxon  is  or  is  not  relevant  to  the  measurement  issue  is  presented  there. 
Similar  (but  undocumented)  examinations  were  made  of  the  process  and  product 
taxa.  The  final  conclusion  of  these  examinations  was  that  119  of  the  281  taxa 
are  relevant  to  evaluation  of  the  Design  Subsystem's  specifications  of  learner 
testing.  Those  relevant  taxa  form  14  taxonomic  subsets,  of  which  5  are 
potentialities,  3  are  processes,  and  6  are  products.  Figure  13  arrays  these 
subsets,  displaying  their  member  taxa  as  issues  relevant  to  evaluation  of 
learner  testing  specifications. 

4.  Addressing  the  Issues:  Attributes  and  Measures  of 

the  Adequacy  of  Gunner -Learner  Testing  SpecificaTIons 

Alongside  each  of  the  evaluative  issues  arrayed  in  Figure.  13,  the 
reader  will  find  a  set  of  attributes  deemed  to  be  relevant  to  the  particular 
issue.  Referring  to  the  overall  APM  (Figure  1),  the  issues  listed  in  Figure  13 
derive  from  Blocks  1-3,  whereas  their  correlated  attributes  for  potential 
measurement  are  the  result  of  carrying  out  the  Block  6  operations.  Each 
attribute  is  a  concrete  element  of  the  Design  subsystem's  work.  It  is 
something  that  can  be  seen,  examined,  submitted  to  analysis,  taken  apart,  etc. 
In  short,  each  attribute  is  something  that  is  measurable.  Further,  each 
attribute  is  something  that,  when  measured,  can  supply  a  portion  of  the 
answer  to  the  question  posed  in  the  evaluative  issue. 

Some  examples  may  help  to  clarify  the  relationship  between  issues 
and  attributes.  Consider  first  the  final  issue  in  Subset  No.  1  (Figure  13), 
namely,  "Can  the  BIFV  Gunnery  Training  Design  subsystem  insure  that  the 
training  goals  will  be  stated  in  terms  that  imply  the  gunner  behaviors 
required,  so  that  the  testing  specifications  ultimately  may  address  those 
behaviors?"  This  issue  arises  from  a  characteristics-level  potentiality,  i.e., 
the  potential  for  stating  goals  in  behavioral-implication  terms.  That  is  a 
particularly  important  capability  for  test  specification.  Each  gunnery  test  will 
require  the  gunner-learners  to  manifest  some  behavior,  for  instance  a  behavior 
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PART  It  ISSUES  AND  ATTRIBUTES  OF  POTENTIALITIES 


Subset  No.  1:  'Identifying  Goals  snd  Priorities* 


attributes 


Does  the  system  have  the  ability 
to  insure  that  all  gunnery  train¬ 
ing  goals  will  be  identified  so 
that  they  may  be  addressed  in 
gunner  testing  specifications? 


1.1.1  Can  the  system  insure  that  the 
total  scope  of  learning  required 
for  each  gunnery  training  goal 
will  be  defined  so  that  the  full 
scope  of  each  goal  may  be  covered 
in  the  testing  specifications? 

1.1. 1.1  Can  the  system  insure  that  all 
achievements  constituting  the 
scope  of  BIFV  gunnery  learning 
will  be  identified  so  that  the 
testing  specifications  may 
address  each  achievement? 

1.1.2  Can  the  system  insure  that  the 
ultimate  intended  outcomes  of 
BIFV  gunnery  learning  can  be 
stated  so  that  appropriate 
tests  of  those  outcomes  may  be 
specified? 


1.1. 2.1 


Can  the  system  insure  that  the 
necessary  levels  of  achievement 
for  acceptable  BIFV  gunnery  can 
be  determined,  so  that  the  spe¬ 
cified  tests  may  assess  whether 
whether  those  levels  are  reached? 


1.1. 2. 2  Can  the  system  insure  that  candi¬ 
date  gunners'  current  levels  of 
achievement  can  be  tested  to  help 
determine  learning  needs? 


Can  the  system  insure  that  the 
goals  will  be  analysed  to  produce 
a  basis  for  specifying  BIFV  gun¬ 
nery  performance  objectives? 


1.1. 4.1  Does  the  system  propose  to  use  an 
appropriate  taxonomy  for  classi¬ 
fying  goals  in  a  fashion  condu¬ 
cive  to  establishing  correct  per¬ 
formance  objectives  and  corres¬ 
pondingly  correct  testing 
specifications? 

1.1. 4. 2  Can  the  system  insure  that  goals 
will  be  stated  in  terms  that 
imply  the  gunner  behaviors  re¬ 
quired,  so  thst  the  testing  spe¬ 
cifications  ultimately  may 
address  those  behaviors? 


Scope  of  information  available  concern¬ 
ing  BIFV  gunnery  operations 
Staff  experience  and  qualifications  in 
the  identification  of  training  goals 
The  planned  approach  to  identifying 
BIFV  gunnery  training  goals 
The  resources  allocated  to  identifying 
BIFV  gunnery  training  goals 

The  elements  or  aspects  of  each  gun¬ 
nery  training  goal  intended  to  be 
defined 

The  plans  for  defining  the  total  scope 
of  BIFV  gunnery  learning 


The  types  or  classes  of  achievement 
Intended  to  be  identified 
The  plans  for  identifying  all  types 
of  BIFV  gunnery  achievements 


Dements  of  ultimate  outcomes  that 
are  intended  to  be  stated 
Intended  formats  for  stating  the  ulti¬ 
mate  intended  outcomes  of  BIFV 
gunnery  learning 
The  plans  for  stating  the  ultimate 
intended  outcomes 

Dements  or  factors  of  the  necessary 
levels  intended  to  be  determined 
(i.e.,  factors  of  acceptability) 

The  plans  for  determining  the  neces¬ 
sary  levels  of  BIFV  gunnery  achieve¬ 
ment 

Dements  or  factor*  of  the  current 
levels  of  achievement  intended  to  be 
tested 

The  plans  for  testing  candidates'  cur¬ 
rent  levels  of  BIFV  gunnery  achieve¬ 
ment 

Elements  of  the  basis  intended  to  be 
established 

Flans  for  establishing  the  basis  for 
specifying  BIFV  gunnery  performance 
objectives 

Goal  taxa  intended  to  be  employed 
Plans  for  employing  the  taxa  to 
classify  goals  for  BIFV  gunnery 
training 


Behavioral  terminology  intended  to  be 
used  in  stating  BIFV  gunnery  training 

goill 

Intended  formats  for  stating  BIFV 
gunnery  training  goals 


Figure  13.  Measurement  Issues  and  Measurable  Attributes  Relevant  to  Evaluation 

of  BIFV  Gunnery  Testing  Specifications 
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1.2  Doe*  the  system  have  the  ability 
to  insure  that  all  B1FV  gunnery 
performance  objectives  will  be 
identified  so  that  they  may  be 
addressed  in  gunner  testing  spe¬ 
cifications? 


1.2.1  Can  the  system  insure  that  all 
things  BIFV  gunners  will  be  able 
to  do  after  training  will  be 
stated,  so  that  tests  of  all 
required  abilities  may  be 
specified? 

1.2. 1.1  Can  the  system  insure  that  the 
stated  abilities  will  include 
definitions  of  the  required 
gunner  actions,  so  that  the  test 
specifications  may  addreas  thoae 
actions? 

1.2. 1.2  Can  the  system  insure  that  the 
stated  abilities  will  Include 
definitions  of  the  appropriate 
conditions  under  which  BIFV  gun¬ 
ners  are  to  perform  the  actions, 
so  that  the  test  specification* 
may  reflect  those  conditions? 

1.2. 1.3  Can  the  system  insure  that  the 
stated  abilities  will  include 
definitions  of  the  appropriate 
standards  of  acceptable  perfor¬ 
mance  of  the  BIFV  gunner 
actions,  so  that  the  test  speci¬ 
fications  may  incorporate  those 
standards? 

1.2.2  Can  the  system  insure  that  the 
identified  BIFV  gunnery  perfor¬ 
mance  objectives  will  be  applied 
to  provide  a  basis  for  objec¬ 
tively  assessing  gunners'  per¬ 
formances? 

1.2.2. 1  Can  the  system  insure  that 
appropriate  pretest  items 
addressing  the  performance  ob¬ 
jectives  will  be  developed,  so 
that  they  stay  be  Included  in  the 
testing  specifications? 

1.2. 2. 2  Can  the  system  Insure  that 
appropriate  posttest  items 
addressing  the  performance  ob¬ 
jectives  will  be  developed,  so 
that  they  may  be  included  in 
the  testing  specifications? 


Figure  13. 


°  Scope  of  Information  available  con¬ 
cerning  BIFV  gunners'  tasks 
0  Staff  experience  and  qualifications 
in  establishing  performance  objectives 
°  The  planned  approach  to  establishing 
BIFV  gunnery  performance  objectives 
°  The  resources  allocated  to  establish¬ 
ing  BIFV  gunnery  performance  objec¬ 
tives 

°  The  elements  or  factor*  of  gunner 
abilities  intended  to  be  stated 
°  Intended  formats  for  stating  BIFV 
gunner  abilities 

0  The  plans  for  stating  intended 
abilities 

°  The  types  of  performance  actions 
intended  to  be  defined 
°  The  plans  for  defining  actions 
required  of  BIFV  gunners 


°  The  elements  or  factors  of  perfor¬ 
mance  condition  intended  to  be 
defined 

°  The  plans  for  defining  BIFV  gunnery 
performance  conditions 


0  Elements  or  factors  of  performance 
standards  Intended  to  be  defined 
0  The  plans  for  defining  standards  for 
BIFV  gunnery  performance  actions 


°  Elements  of  the  basis  for  assessment 
intended  to  be  provided 
°  Plans  for  providing  a  basis  for  objec¬ 
tively  assessing  BIFV  gunners' 
performances 


0  Types  of  BIFV  gunnery  pretest  items 
intended  to  be  developed 
°  Plans  for  developing  BIFV  gunnery 
.  pretest  items 


°  Types  of  BIFV  gunnery  posttest  items 
intended  to  be  developed 
°  Plans  for  developing  BIFV  gunnery 
posttest  items 


(Continued) 
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Subget  No.  3i  "Analysing  Performance  Objectives" 
Issues  — 


Attribute! 


1.3  Does  the  system  hsve  the  ability 
to  insure  that  all  BIFV  gunnery 
performance  objectives  will  be 
analysed  to  determine  the  spe¬ 
cific  elements  of  gunner  perfor¬ 
mance  that  need  to  be  tested? 


1.3.1  Can  the  system  insure  that  the 
analysis  of  BIFV  gunnery  perfor¬ 
mance  objectives  will  disclose 

the  gunnery  training  prerequi¬ 
sites,  so  that  appropriate  testa 
of  candidate  gunners  may  be 
specified? 

1.3. 1.1  Can  the  system  insure  that  the 
analysis  will  uncover  what  can¬ 
didate  gunners  must  recall  in 
order  to  be  qualified  for  train¬ 
ing,  so  that  appropriate  tests 
of  that  recall  may  be  specified? 

1.3. 1.2  Can  the  system  insure  that  the 
analysis  will  uncover  abilities 
that  the  candidate  gunners  abso¬ 
lutely  must  posaeas  in  order  to 
achieve  the  performance  objec¬ 
tives,  ao  that  appropriate  tests 
of  those  abilities  may  be 
specified? 

1.3. 1.3  Can  the  system  insure  that  the 
analysis  aril]  uncover  abilities 
which,  if  possessed  by  candidate 
gunners,  will  facilitate 
achievement  of  the  performance 
objectives,  so  that  appropriate 
tests  of  those  abilities  may  be 
specified? 

1.3.2  Can  the  system  insure  that  the 
analysis  of  performance  objec¬ 
tives  will  be  applied  to  provide 
a  basis  for  objectively  assess¬ 
ing  a  candidate  gunner's  suit¬ 
ability  for  training? 

1.3. 2.1  Can  the  system  insure  that 
appropriate  entry  test  items 
addressing  candidate  BIFV  gun¬ 
ners'  suitability  will  be 
included  in  the  testing  speci¬ 
fications? 

1.3.4  Can  the  system  insure  that  per¬ 
formance  objectives  will  be 
classified  into  appropriate 
domains  of  learning,  so  that 
testa  specified  for  each  objec¬ 
tive  may  exercise  the  appro¬ 
priate  domain? 

Figure  13. 


0  Intended  acope  of  analysis  of  BIFV 
gunnery  performance  objectives 
°  Staff  experience  and  qualifications 
in  analysing  performance  objectives 
0  The  planned  approach  to  analysing 
BIFV  gunnery  performance  objectives 
0  The  resources  allocated  to  analysing 
BIFV  gunnery  performance  objectives 

°  Types  or  classes  of  BIFV  gunnery 
prerequisites  intended  to  be 
identified 

°  Plans  for  identifying  BIFV  gunnery 
training  prerequisites 


°  Types  of  information  recall  require¬ 
ments  intended  to  be  revealed 
°  Plans  for  revealing  the  BIFV  gun¬ 
ners'  recall  requirements 


0  Types  of  essential  prerequisite 
abilities  intended  to  be  revealed 
°  Plans  for  revealing  the  BIFV  gun¬ 
ners'  essential  prerequisite 
abilities 


0  Types  of  supportive  prerequisite 
abilities  intended  to  be  revealed 
0  Plans  for  revealing  the  BIFV  gun¬ 
ners'  supportive  prerequisite 
abilities 


0  Elements  of  the  basis  for  candidate 
assessment  intended  to  be  provided 
°  Plans  for  providing  a  basis  for 
objectively  assessing  a  candidate 
BIFV  gunner's  suitability  for 
-training 

0  Scope  of  BIFV  gunner  abilities 
intended  for  entry  testing 
°  Plans  for  specifying  entry  test  items 
for  candidate  BIFV  gunners 


°  Domains  of  learning  intended  for  use 
in  classifying  BIFV  gunnery  perfor¬ 
mance  objectives 

0  Plans  for  classifying  BIFV  gunnery 
performance  objectives  into  learning 
domains 
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Sublet  No.  3  (Cont'd) _ _ 

Issues  "  Attributes 


1.3.4. 1  Can  the  system  insure  that  all 
information-type  objectives 
trill  be  correctly  classified. 

so  that  appropriate  informa¬ 
tion-type  tests  may  be  speci¬ 
fied  for  them? 

1.3. 4. 2  Can  the  system  insure  that  all 
mental  skills-type  objectives 
will  be  correctly  classified, 

so  that  appropriate  mental 
skills-type  tests  may  be  speci¬ 
fied  for  them? 

1.3. 4. 3  Can  the  system  insure  that  all 
physical  skills-type  objectives 
will  be  correctly  classified,  so 
that  appropriate  physical 
akitla-type  tests  may  be  speci¬ 
fied  for  them? 

1.3. 4. 4  Can  the  system  insure  that  all 
attitude-type  objectives  will  be 
correctly  classified,  so  that 
appropriate  attitude-type  tests 
may  be  specified  for  them? 

1.3.5  Can  the  system  insure  that  the 

analysis  of  BIFV  gunnery  perfor¬ 
mance  objectives  will  identify 
the  component  steps  or  processes 
required  of  BIFV  gunners,  so 
that  tests  of  the  gunners'  abil¬ 
ities  to  perform  those  component 
steps  or  processes  may  be 
specified? 

1.3.5. 1  Can  the  system  insure  that  all 
overt  steps  required  of  BIFV 
gunners  will  be  identified,  so 
that  they  may  be  addressed  in 
the  testing  specifications? 

1.3. 5. 2  Can  the  system  insure  that  all 
covert  stops  required  of  BIFV 
gunners  will  be  identified,  so 
that  they  may  be  addressed  in 
the  testing  specifications? 

1.3. 5. 3  Can  the  system  insure  that  all 
unconscious  steps  required  of 
BIFV  gunners  will  be  identified, 
so  that  they  may  be  addressed  in 
the  testing  specifications? 


0  Criteria  intended  to  be  employed  for 
classifying  BIFV  gunnery  performance 
objectives  as  information-type 
objectives 


0  Criteria  intended  to  be  employed  for 
classifying  BIFV  gunnery  performance 
objectives  as  mental  skills-type 
objectives 


°  Criteria  intended  to  be  employed  for 
classifying  BIFV  gunnery  performance 
objectives  as  physical  skills-type 
objectives 


°  Criteria  intended  to  be  employed  for 
classifying  BIFV  gunnery  performance 
objectives  as  attitude-type  objec¬ 
tives 


°  Types  or  classes  of  gunners'  compo¬ 
nent  steps  or  processes  intended  to 
be  identified 

0  Plans  for  identifying  gunners'  com¬ 
ponent  steps  or  processes  within  the 
BIFV  gunnery  performance  objectives 


°  Types  of  overt  steps  intended  to  be 
identified 

°  Plans  for  identifying  the  overt 
steps  involved  in  BIFV  gunnery 
performance  objectives 

°  Types  of  covert  steps  intended  to  be 
identified 

°  Plans  for  identifying  the  covert 
steps  involved  in  BIFV  gunnery 
performance  objectives 

°  Types  of  unconscious  steps  Intended 
to  be  identified 

°  Plans  for  identifying  the  unconscious 
steps  involved  in  BIFV  gunnqry 
performance  objectives 
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1.5  Does  the  system  hsve  the  ability 
to  Insure  that  all  BIFV  gunnery 
training  procedures  will  be  cor¬ 
rectly  defined,  so  that  gunnery 
testing  procedures  will  play 
their  proper  role  in  BIFV 
gunnery  training? 


1.5.3  Can  the  system  insure  that  BIFV 
gunner  application  (or  practice) 
procedures  will  be  correctly 
specified,  so  that  the  role  of 
testing-in-practice  will  be 
properly  defined  and  reflected 
in  the  testing  specifications? 

1.5. 3. 4  Can  the  system  insure  that  the 
application  (practice)  pro¬ 
cedures  will  provide  learning 
feedback  to  the  BIFV  gunner- 
learners? 


1.5. 3. 5  Can  the  system  insure  that  the 
application  (practice)  pro¬ 
cedures  will  enable  the  BIFV 
gunner-learners  to  assess 
sufficiency  of  practice? 


1.5.4  Can  the  system  insure  that  BIFV 
gunner  evaluation  (or  testing) 
procedures  will  be  correctly 
defined,  so  that  those  defini¬ 
tions  may  be  reflected  in  the 
testing  specifications? 

1.5. 4.1  Can  the  system  insure  that  the 
BIFV  gunner  tests  that  are  spe¬ 
cified  will  address  the  appro¬ 
priate  domains  of  BIFV  gunnery 
learning? 

1.5. 4. 2  Can  the  system  insure  that  the 
BIFV  gunner  tests  that  are  spe¬ 
cified  will  elicit  the  appro¬ 
priate  actions  from  the  BIFV 
gunner-learners? 

1.5. 4. 3  Can  the  system  insure  that  the 
BIFV  gunner  tests  that  are  spe¬ 
cified  will  provide  a  reliable 
assessment  of  the  BIFV  gunner- 
learners'  performances? 


°  Categories  or  types  of  BIFV  gunnery 
training  procedures  intended  to  be 
defined 

°  Staff  experience  and  qualifications 
in  the  definition  of  training 
procedures 

0  The  planned  approach  to  defining 
BIFV  gunnery  training  procedures 
°  The  resources  allocated  to  defining 
BIFV  gunnery  training  procedures 

0  Classes/methods  of  BIFV  gunnery 
application  intended  to  be  specified 
0  Intended  involvement  of  testing  in 
■  BIFV  gunnery  application 
0  Plans  for  specifying  BIFV  gunnery 
application  procedures 


°  Intended  role  of  learning  feedback 
in  the  BIFV  gunnery  application 
procedures 

°  Intended  contribution  of  testing  to 
BIFV  gunnery  learning  feedback 
°  Plans  for  providing  for  BIFV 
gunnery  learning  feedback 

0  Criteria  intended  to  be  employed  in 
assessing  sufficiency  of  practice 
°  Intended  contribution  of  testing 
to  the  employment  of  those  criteria 
°  Plans  for  enabling  BIFV  gunner- 
learners  to  assess  sufficiency  of 
practice 

0  Categories/methods  of  BIFV  gunnery 
evaluation  intended  to  be  specified 
°  Plans  for  specifying  BIFV  gunnery 
evaluation  procedures 


°  Plans  for  tailoring  test  specifica¬ 
tions  to  the  domain  of  the  objective 
to  be  tested 


°  Plans  for  tailoring  test  specifica¬ 
tions  to  the  action  called  for  in 
the  objective  to  be  tested 


°  Plans  for  tailoring  test  specifica¬ 
tions  to  the  conditions  and  stand¬ 
ards  appropriate  to  the  objective  to 
be  tested 

6  Plans  for  devising  test  specifica¬ 
tions  that  will  support  statistical 
reliability  requirements 


Figure  13.  (Continued) 


1.6  Does  the  system  have  the  ability 
to  insure  that  the  BIFV  gunnery 
training  curriculum  will  be 
properly  evaluated,  and  in  par¬ 
ticular  that  all  BIFV  gunner 
testing  appropriate  to  curricu¬ 
lum  evaluation  will  be  specified 
and  conducted? 


1.6.1  Can  the  system  insure  that  all 
appropriate  steps  will  be  taken 
to  vslidate/revise  the  bases  for 
the  developing  BIFV  gunnery 
training  curriculum,  including 
in  particular  the  specification 
and  conduct  of  BIFV  gunner  test¬ 
ing  appropriate  to  that 
validation  /revision  ? 

1.6. 1.1  Can  the  system  insure  that  all 
appropriate  entry-level  testing 
of  representative  candidate  BIFV 
gunners  will  be  specified  and 
conducted  to  verify  assumptions 
concerning  prerequisites  and 
candidates'  qualifications? 


1.6. 1.2  Can  the  system  insure  that  all 
appropriate  post-training  test¬ 
ing  of  representative  current 
BIFV  gunners  will  be  specified 
and  conducted  to  verify  conclu¬ 
sions  concerning  the  performance 
objectives  and  the  tests  speci¬ 
fied  for  those  objectives? 

1.6.2  Can  the  system  insure  that  all 
appropriate  steps  will  be  taken 
to  validate /revise  the  develop¬ 
ing  BIFV  gunnery  training  cur¬ 
riculum  itself,  including  in 
particular  the  specification  and 
conduct  of  BIFV  gunner  testing 
appropriate  to  that  validation/ 
revision? 

1.6. 2.1  Can  the  system  insure  that  all 
appropriate  testing  of  "pilot 
training"  BIFV  gunner-learners 
will  be  specified  and  conducted, 
so  that  necessary  error  data  on 
the  specified  tests  may  be 
obtained? 

1.6. 2. 2  Can  the  system  insure  that 
"pilot  training*  gunner-learners' 
performance  on  the  specified 
tests  will  be  correlated  with 
subsequent  ]ob  performance 
measures,  so  that  the  predictive 
validity  of  the  specified  tests 
may  be  assessed? 


°  Intended  scope  of  BIFV  gunnery 
training  curriculum  evaluation 
°  Staff  experience  and  qualifications 
in  evaluation  of  training  curricula 
0  The  planned  approach  to  evaluating 
the  BIFV  gunnery  training  curriculum 
°  The  resources  allocated  to  evaluat¬ 
ing  the  BIFV  gunnery  training 
curriculum 

°  Intended  elements /methods  of  cur¬ 
riculum  basis  validation/revision 
to  be  employed 

°  Intended  contribution  of  BIFV  gunner 
testing  to  those  elements /methods 
°  Plans  for  validating /revising  the 
bases  for  the  developing  BIFV 
gunnery  training  curriculum 


°  Assumptions  concerning  prerequisites 
and  candidates'  qualifications 
intended  to  be  verified 
°  Intended  contribution  of  BIFV  entry- 
level  gunner  testing  to  that 
verification 

°  Plans  for  specifying  and  conducting 
BIFV  entry-level  gunner  testing  to 
verify  the  assumptions 

°  Conclusions  concerning  performance 
objectives  and  test  specifications 
intended  to  be  verified 
°  Intended  contribution  of  BIFV  post¬ 
training  testing  to  that  verification 
0  Plans  for  specifying  and  conducting 
BIFV  post-training  testing  to  verify 
the  conclusions 

°  Intended  elements/methods  of  curricu¬ 
lum  validation /revision  to  be  em¬ 
ployed 

°  Intended  contribution  of  BIFV  gunner 
testing  to  those  elements/methods 
0  Plans  for  validating /revising  the 
developing  BIFV  gunnery  training 
curriculum 


°  Aspects /segments  of  the  curriculum 
intended  for  "pilot  training" 

°  Intended  role  of  BIFV  gunner  testing 
in  "pilot  training" 

°  Plans  for  specifying  and  conducting 
BIFV  gunner  testing  in  conjunction 
with  "pilot  training* 

°  Aspects /segments  of  the  curriculum 
for  which  job  performance  measures 
are  intended  to  be  developed 
0  Plans  for  correlating  job  perfor¬ 
mance  measures  with  "pilot  training" 
gunner-learners'  test  scores 


Figire  IS.  (Continued) 


PART  2:  ISSUES  AND  ATTRIBUTES  OF  PROCESSES 


2.3  Does  the  system  conduct  s  proper  0  The  task  analytic  methods  applied  to 

analysis  of  BIFV  gunner  tasks,  the  B1FV  gunner  tasks 

so  that  appropriate  tests  of  °  The  scope  of  the  task  analyses  con- 

those  tasks  may  be  specified?  ducted  (i.e. ,  factors  determined  In 

the  analyses) 

°  The  completeness  of  the  BIFV  gunner 
task  analyses 

0  The  accuracy  of  the  BIFV  gunner  task 
analyses 


2.3.2  Does  the  system  accurately  iden¬ 
tify  the  domains  of  learning 
involved  in  the  BIFV  gunner 
tasks,  so  that  tests  may  be  spe¬ 
cified  and  developed  in  the 
proper  domains? 


0  Methods  employed  to  Identify  the 
domains  of  learning  involved  in  the 
BIFV  gunner  tasks 
°  The  domains  of  learning  identified 
for  each  BIFV  gunner  task 


2. 3. 2.1  Does  the  system  accurately  iden-  °  Methods  employed  to  identify  the 

tify  the  knowledge  that  BIFV  the  knowledge  requirements  associ- 

gunners  must  acquire  to  perform  ated  with  BIFV  gunner  tasks 

their  tasks,  so  that  appropriate  °  The  BIFV  gunner  task  knowledge 

tests  of  knowledge  may  be  speci-  requirements  identified 

fied  and  developed? 


2. 3. 2. 2  Does  the  system  accurately  iden¬ 
tify  the  mental  skills  that  BIFV 
gunners  must  acquire  to  perform 
their  tasks,  so  that  appropriate 
tests  of  mental  skills  may  be 
specified  and  developed? 


•  Methods  employed  to  identify  the 
mental  skills  requirements  associ¬ 
ated  with  BIFV  gunner  tasks 
“  The  BIFV  gunner  task  mental  skills 
requirements  identified 


2. 3. 2. 3  Does  the  system  accurately  iden¬ 
tify  the  physical  skills  that 
BIFV  gunners  must  acquire  to 
perform  their  tasks,  so  that 
appropriate  tests  of  physical 
skills  may  be  specified  and 
developed? 

2. 3. 2. 4  Does  the  system  accurately  iden¬ 
tify  the  attitudes  that  BIFV 
gunners  must  acquire  to  perform 
their  tasks,  so  that  appropriate 
tests  of  attitude  may  be  speci¬ 
fied  and  developed? 

2.3.3  Does  the  system  sccurately  iden¬ 
tify  the  conditions  for  learning 
associated  with  the  BIFV  gunner 
tasks,  so  that  appropriate  tests 
may  be  specified  and  developed 
for  assessing  candidate  gunners' 
qualifications? 

2. 3. 3.1  Does  the  system  accurately  iden¬ 
tify  BIFV  gunners'  prerequisite 
knowledge,  so  that  appropriate 
tests  of  the  knowledge  prerequi¬ 
sites  stay  be  specified  and 
developed? 


0  Methods  employed  to  identify  the 
physical  skills  requirements  associ¬ 
ated  with  BIFV  gunner  tasks 
0  The  BIFV  gunner  task  physical  skills 
requirements  identified 


°  Methods  employed  to  identify  the 
attitudinal  requirements  associated 
with  BIFV  gunner  tasks 
*  The  BIFV  gunner  task  attitudinal 
requirements  identified 


•  Methods  employed  to  identify  the 
conditions  for  learning  the  BIFV 
gunner  tasks 

*  Types  of  conditions  for  learning 
identified  for  each  BIFV  gunner  task 


*  Methods  employed  to  identify  the  pre¬ 
requisite  knowledge  required  for 
learning  the  BIFV  gunner  tasks 

*  Knowledge  items  identified  as  pre¬ 
requisite  for  learning  the  BIFV 
gunner  tasks 
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Subget  No.  6  (Cont'd) 
Issues 


Attributed 


2. 3. 3. 2  Doe*  the  system  accurately  Iden¬ 
tify  BIFV  funner*'  prerequisite 
mental  skills,  so  that  appropri¬ 
ate  tests  of  the  mental  akilla 
prerequisites  may  be  specified 
and  developed? 

2. 3. 3. 3  Does  the  system  accurately  iden¬ 
tify  BIFV  gunners'  prerequisite 
physical  skills,  so  that  appro¬ 
priate  tests  of  the  physical 
skills  prerequisites  may  be  spe¬ 
cified  and  developed? 

2. 3. 3. 4  Does  the  system  accurately  iden¬ 
tify  BIFV  gunners'  prerequisite 
attitudes,  so  that  appropriate 
tests  of  the  attitudinal  pre¬ 
requisites  may  be  specified  and 
developed? 


0  Methods  employed  to  identify  the  pre¬ 
requisite  mental  skills  required  for 
learning  the  BIFV  gunner  tasks 
*  Specific  mental  skills  identified  as 
prerequisite  for  learning  the  BIFV 
gunner  tasks 

0  Methods  employed  to  identify  the  pre¬ 
requisite  physical  skills  required 
for  learning  the  BIFV  gunner  tasks 
0  Specific  physical  skills  identified 
as  prerequisite  for  learning  the 
BIFV  gunner  tasks 

°  Methods  employed  to  identify  the  pre¬ 
requisite  attitudes  required  for 
learning  the  BIFV  gunner  tasks 
°  Specific  attitudes  identified  as  pre¬ 
requisite  for  learning  the  BIFV 
gunner  tasks 


Figure  13  (Continued) 
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2.4  Given  the  results  of  the  analy- 
•is  of  the  tasks  selected  for 
training,  does  the  system  propr 
erly  determine  the  BIFV  gunnery 
instructional  requirements, 
including  in  particular  the 
requirements  for  testing  gunner- 
learners? 


2.4.1  Does  the  system  accurately  iden¬ 
tify  all  required  BIFV  gunnery 
learning  activities/events,  so 
that  the  proper  role  of  testing 
in  those  activities/events  may 
be  specified? 

2. 4. 1.4  Doee  the  system  accurately  iden¬ 
tify  the  gunner  testing  compo¬ 
nents  of  the  required  BIFV 
gunnery  learning  activities/ 
events? 


2.4.4  Does  the  system  accurately  iden¬ 
tify  all  needs  for  testing  BIFV 
gunner-learners  in  conjunction 
with  BIFV  gunnery  training? 
(This  is  in  addition  to  needs 
for  testing  as  component  parts 
ol  BIFV  gunnery  learning  activi¬ 
ties  /events. ) 


2.4.4. 1  Does  the  system  accurately  iden¬ 
tify  all  needs  for  testing 
candidates  for  eccer-tnce  as 
BIFV  gunner-learners? 


2. 4. 4. 2  Does  the  system  accurately  iden¬ 
tify  all  needs  for  testing 
accepted  BIFV  gunner-learners  to 
assess  their  requirements  for 
remedial  training? 


2. 4. 4. 3  Does  the  system  accurately  iden¬ 
tify  all  needs  for  testing 
accepted  BIFV  gunner-learners  to 
assess  possible  requirements  for 
tailoring  the  training  to  their 
abilities  and  experience? 


Figure  13. 


°  The  methods  employed  to  determine 
instructional  requirements  from  the 
results  of  the  analysis  of  BIFV 
gunner  tasks 

•  The  types  of  instructional  require¬ 
ments  determined 

0  The  completeness  of  the  methods' 
applications 

°  The  accuracy  of  the  methods'  appli¬ 
cations 

6  The  methods  employed  to  identify 
required  BIFV  gunnery  learning 
activities  /  events 

°  The  BIFV  gunnery  learning  activi¬ 
ties/events  identified  as  required 


Methods  employed  to  identify  compo¬ 
nents  of  gunner  testing  included 
within  required  BIFV  gunnery  learn¬ 
ing  activities /events 
The  gunner  testing  identified  as 
components  of  the  required  BIFV  gun¬ 
nery  learning  activities /events 

The  methods  employed  to  Identify 
requirements  for  BIFV  gunnery  test¬ 
ing  in  conjunction  with  BIFV 
gunnery  training 

The  types  of  gunnery  testing  needs 
identified 


Methods  employed  to  identify 
acceptance-testing  requirements 
Identified  needs  for  acceptance- 
testing  of  BIFV  gunner-learner 
candidates 

Methods  employed  to  identify  testing 
needed  to  assess  remedial  training 
requirements 

Identified  needs  for  testing 
accepted  BIFV  gunner-learners  to 
assess  their  remedial  training 
requirements 

Methods  employed  to  identify  testing 
needed  to  assess  possible  training 
tailoring  requirements 
Identified  needs  for  testing 
accepted  BIFV  gunner- learn  era  to 
assess  possible  training  tailoring 
required  for  them 
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Subset  No.  7:  (Copt'd? _ 

Issues*  ~ Attributes 


2. 4. 4. 4  Does  the  system  accurately  Iden¬ 
tify  all  needs  for  testing  BIFV 
gunner-learners  during  training 
to  assess  requirements  for 
advancing  or  delaying  their 
progress  through  the  training? 

2. 4. 4. 5  Does  the  system  accurately  iden¬ 
tify  all  needs  for  testing  BIFV 
gunner-learners  at  completion  of 
scheduled  training  to  assess 
their  qualifications  for  cer¬ 
tification  as  BIFV  gunners? 


Methods  employed  to  identify  testing 
needed  to  assess  learner  advancement 
requirements 

Identified  needs  for  testing  BIFV 
gunner-learners  to  assess  their 
training  advancement  status 

Methods  employed  to  identify  gunner 
certification  testing  requirements 
Identified  needs  for  testing  BIFV 
gunner-learners  to  determine  their 
certification  status 


Figure  13.  (Continued) 


Subset  No.  8:  ‘Conduct  Training  Trials" 

Issues 


Attributes 


2.7  Does  the  system  appropriately 
conduct  trials  of  BIFV  gunnery 
training,  including  in  particu¬ 
lar  trials  of  gunnery  testing? 


2.7.2  Does  the  system  adequately  and 
accurately  verify  the  suitabil¬ 
ity  of  the  assembled  instruction 
for  delivery,  including  in  par¬ 
ticular  the  testing  segments  of 
instruction? 


2.7. 2.1  Does  the  system  conduct  appro¬ 
priate  field  tests  of  the  assem¬ 
bled  BIFV  gunnery  instruction, 
including  field  tests  of  the 
testing  specifications? 


0  The  aspects/segments  of  BIFV  gunnery 
training  submitted  to  trial 

0  The  kinds  of  trials  conducted 

°  The  role  of  gunner-learner  testing 
in  the  trials 

°  The  appropriateness  of  the  trials 

°  The  completeness  of  the  trials 

0  The  reliability  of  the  trials 

°  Methods  employed  to  verify  the  suit¬ 
ability  of  the  BIFV  gunnery  instruc¬ 
tion  for  delivery 

°  The  role  of  gunner-learner  testing 
in  those  methods 

0  Conclusions  reached  concerning  the 
suitability  for  delivery 

0  Kinds  of  field  tests  conducted  of 
the  assembled  BIFV  gunnery  instruc¬ 
tion 

°  Information  gleaned  from  field  tests 

°  The  role  of  gunner-learner  testing 
in  the  field  tests 


Figure  13.  (Continued) 


Subset  No.  9: 


‘Documented  Job  Analyses" 
Issues 


3.1 


Does  the  system  produce  docu¬ 
mented  analyses  of  the  BIFV  gun¬ 
ner's  Jobs  that  completely 
define  the  performance  elements 
essential  for  adequate  gunnery, 
so  that  tests  of  those  elements 
may  be  specified? 


3.1.1 


3. 1.1.1 


3. 1.1. 2 


3.1.3 


Does  the  system  produce  a  com¬ 
plete  specification  of  the  per¬ 
formance  elements  of  the  BIFV 
gunner's  jobs  in  sufficient 
detail  to  permit  those  elements 
to  be  tested? 


Does  the  system  produce  accurate 
specifications  of  all  BIFV  gun¬ 
ner  mental  processes  to  the 
level  of  binary  decisions,  so 
that  tests  of  those  decisions 
may  be  constructed? 


Does  the  system  produce  accurate 
specifications  of  all  BIFV  gun¬ 
ner  physical  processes  to  the 
level  of  discrete  actions,  so 
that  tests  of  those  actions  may 
be  specified? 


3. 1-1. 3 


Does  the  system  produce  accurate 
specifications  of  the  stimuli 
for  all  BIFV  gunner  procesaes, 
so  that  those  stimuli  may  be 
incorporated  into  the  tests  of 
those  processes? 


3. 1.1. 4 


Does  the  system  produce  accurate 
specifications  of  the  conditions 
under  which  BIFV  gunners  perform 
their  processes,  so  that  those 
conditions  may  be  incorporated 
into  the  tests  of  those 


processes? 


Does  the  system  produce  a  com¬ 
plete  specification  of  the  fac¬ 
tors  that  the  BIFV  gunner- 
learners'  performance  elements 
must  satisfy  for  adequate 


gunnery? 


3. 1.3.1 


Does  the  system  produce  correct 
specifications  of  the  accuracy 
requirements  associated  with 
BIFV  gunners'  performance,  so 
that  tests  can  be  designed  to 
assess  whether  sufficient  accu¬ 
racy  is  achieved? 


3. 1.3. 2 


Does  the  system  produce  correct 
specifications  of  the  speed 
requirements  associated  with 
BIFV  gunners'  performance,  so 
that  tests  can  be  designed  to 
assess  whether  sufficient  speed 
ts  achieved? 


Attributes 


The  BIFV  gunner  job  analyses  docu¬ 
mented 

The  elements  of  BIFV  gunner  perfor¬ 
mance  identified  in  the  documen¬ 
tation 

The  completeness  of  the  documents 
The  accuracy  of  the  documents 


BIFV  gunner  performance  elements 
for  which  detailed  specifications 
are  produced 

Types  of  details  specified  for 
each  performance  element 


BIFV  gunner  mental  processes  speci¬ 
fied  to  the  level  of  binary  deci¬ 
sions 

The  binary  decisions  specified  for 
each  BIFV  gunner  mental  process 


BIFV  gunner  physical  processes  spe¬ 
cified  to  the  level  of  discrete 
actions 

The  discrete  actions  specified  for 
each  BIFV  gunner  phyaical  process 


BIFV  gunner  processes  for  which 
stimuli  are  specified 
The  stimulus  specified  for  each 
BIFV  gunner  process 


BIFV  gunner  processes  for  which 
conditions  are  specified 
The  conditions  specified  for  each 
BIFV  gunner  process 


BIFV  gunner  performance  elements  for 
which  adequacy  factors  are  specified 
Types  of  adequacy  factors  specified 
for  each  performance  element 


BIFV  gunner  processes  for  which 
accuracy  requirements  are  specified 
The  accuracy  requirements  specified 
for  each  BIFV  gunner  process 


BIFV  gunner  processes  for  which 
speed  requirements  are  specified 
The  speed  requirements  specified 
for  each  BIFV  gunner  process 
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3. 1.3.3  Does  the  system  produce  correct 
specifications  of  the  volume 
requirements  associated  with 
BIFV  gunners'  performance,  so 
that  tests  can  be  designed  to 
assess  whether  sufficient 
volume  is  achieved? 


3. 1.3. -4  Does  the  system  produce  correct 
specifications  of  the  duration 
requirements  associated  with 
BIFV  gunners'  performance,  so 
that  tests  can  be  designed  to 
assess  whether  sufficient  dura¬ 
tion  is  achieved? 


0  BIFV  gunner  processes  for  which 
volume  requirements  are  specified 
°  The  volume  requirements  specified 
for  each  BIFV  gunner  process 


0  BIFV  gunner  processes  for  which 
duration  requirements  are  specified 
°  The  duration  requirements  specified 
for  each  BIFV  gunner  process 


Figure  13.  (Continued) 


Sublet  No.  10:  "Stated  Performance  Objectives" 


Does  the  system  produce  accurate 
and  complete  statement*  of  the 
performance  objective*  that  BIFV 
gunners  must  achieve  in  order  to 
be  qualified  for  that  job,  so 
that  testa  of  gunners'  qualifi¬ 
cations  may  be  specified? 


3.2.1  Does  the  system  produce  exact 
definitions  of  what  the  gunner- 
learner  are  expected  to  achieve 
through  training,  so  that  tests 
of  thoe  achievements  may  be 
developed? 

3. 2. 1.1  Does  the  system  produce  correct 
specifications  of  the  capabili¬ 
ties  that  gunner-learners  are 
expected  to  achieve? 


3. 2. 1.2 


Does  the  system  produce  correct 
specifications  of  the  actions 
that  the  gunner-learners  are  to 
execute  to  demonstrate  the 
capabilities? 


3.2. 1.3  Does  the  system  produce  correct 
specifications  of  the  objects 
that  are  to  result  from  the 
execution  of  the  action*  by  the 
gunner-learners? 

3. 2. 1.4  Does  the  system  produce  correct 
specifications  of  the  circum¬ 
stances  under  which  the  gunner- 
learners  are  to  execute  the 
actions  to  produce  the  objects? 

3.2. 1.5  Does  the  system  produce  correct 
specifications  of  the  tools  and 
equipment  BIFV  gunner-learners 
are  to  use  in  executing  the 
actions? 

3. 2. 1.6  Does  the  system  produce  correct 
specifications  of  the  con¬ 
straints  to  be  imposed  upon  the 
execution  of  the  actions  by  the 
gunner-learners? 

3.2. 1.7  Does  the  system  produce  correct 
specifications  of  the  criteria 

to  be  used  to  judge  the  adequacy 
of  the  objects  resulting  from 
the  actions  executed  by  the 
gunner- learn  ere? 


The  BIFV  gunner  performance  objec¬ 
tives  that  are  stated 
The  format  of  the  statements  of 
BIFV  gunner  performance  objectives 
The  completeness  of  the  performance 
objective  statements 
The  accuracy  of  the  performance 
objective  statements 

The  gunner-learner  achievements 
that  are  defined 

The  factors  included  in  the  defini¬ 
tions  of  gunner-learner  achievements 


Capabilities  specified  for  achieve¬ 
ment  by  BIFV  gunner-learners 


Actions  specified  for  execution  by 
BIFV  gunner-learners 
Observability  of  the  actions 


Objects  specified  as  results  of  the 
action  execution*  by  BIFV  gunner- 
learners 

Measurability  of  the  objects 


Circumstances  specified  for  execu¬ 
tion  of  the  actions  by  BIFV  gunner- 
learners 


Tools  and  equipment  specified  for 
use  in  the  execution  of  the  actions 
by  BIFV  gunner- learners 


Constraints  specified  for  the 
execution  of  the  actions  by  the 
BIFV  gunner- Jeamers 


Criteria  specified  as  standards  of 
sdequacy  for  the  objects  resulting 
from  the  gunner-learners'  actions 
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Subset  No.  11:  ‘Lesson  Plans'1 


Does  the  system  produce  adequate 
lesson  plans  outlining  the  con¬ 
tent  and  procedures  for  all 
learning  activities,  including 
gunner-learner  testing  in  con¬ 
junction  with  learning  activi¬ 
ties? 

Does  the  system  produce  complete 
specifications  of  the  instruc¬ 
tional  content  for  the  learning 
activities,  including  the  con¬ 
tent  of  gunner-learner  testing 
in  conjunction  with  learning 
activities? 


3. 3. 1.8 


3. 3. 2.  4 


Does  the  system  produce  correct 
specifications  of  test  problems 
and  exercises  in  the  instruc¬ 
tional  content  for  the  BIFV 
gunnery  learning  activities? 

Does  the  system  produce  complete 
specifications  of  the  instruc¬ 
tional  procedures  for  the  learn¬ 
ing  activities,  including  the 
procedures  for  gunner-learner 
testing  in  conjunction  with 
learning  activities? 

Does  the  system  produce  correct 
specifications  of  testing  pro¬ 
cedures  in  the  Instructional 
procedures  for  the  BIFV  gunnery 
learning  activities? 


Subset  No.  1Z:  ‘Training  Documents‘ 


Does  the  system  produce  all 
necessary  training  documents, 
including  in  particular  docu¬ 
ments  needed  for  BIFV  gunner- 
learner  testing? 


Does  the  system  produce  docu¬ 
ments  providing  all  necessary 
guidance  to  BIFV  gunnery  train¬ 
ing  facilitators,  including 
guidance  for  Implementing 
gunner-learner  testing? 


3. 4. 2.2 


Does  the  system  produce  complete 
sets  of  test  specifications, 
addressing  all  applications  and 
requirements  for  BIFV  gunner- 
learner  testing? 


Attributes 

Lesson  plans  prepared  for  BIFV 
gunnery  learning  activities 
Segments  of  the  plans  outlining 
gunner-learner  testing 
Completeness  of  the  testing  segments 
Appropriateness  of  the  testing  seg¬ 
ments  to  the  learning  activities 

BIFV  gunnery  learning  activities 
for  which  content  is  specified 
in  the  lesson  plans 
Content  elements  specified  for 
each  learning  activity 
Content  elements  pertaining  to 
testing 

Test  problems /exercises  itemized 
in  the  lesson  plan  content 


°  BIFV  gunnery  learning  activities 
fur  which  procedures  are  specified 
in  the  lesson  plans 
a  Procedural  elements  specified 
for  each  learning  activity 
°  Procedural  elements  pertaining 
to  testing 

0  BIFV  gunner-learner  testing 
procedures  itemized  in  the 
lesson  plan  procedures 


Attributes  _ 

°  BIFV  gunnery  training  documents 
prepared 

°  Documents  and  segments  of  documents 
that  are  relevant  to  gunner- 
learner  testing 

0  Completeness  of  the  testing 
documents /segments 

°  Accuracy  of  the  testing  documents/ 
segments 

°  Documented  guidance  for  BIFV 
gunnery  training  facilitators 

°  Guidance  elements  provided  in  the 
documentation 

°  Guidance  elements  pertaining  to 
gunner-learner  testing 

0  Testing  applications/requirements 
for  which  BIFV  gunner-learner  test 
specifications  are  documented 

°  Elements  or  aspects  of  testing 
included  in  the  specifications 
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Subset  No.  13;  "Tests 
Issues 


Attributes' 


3.5  Does  the  system  produce  appro 
priate  tests  for  administration 
to  BIFV  gunner-learners? 


3.5.1  Does  the  system  produce  appro¬ 
priate  tests  for  assessing  the 
qualifications  of  candidate  BIFV 
gunner-learners? 

3. 5. 1.1  Do  the  candidate  qualification 
tests  provide  adequate  measures 
of  the  candidates'  prerequisite 
abilities? 


3. 5. 1.2  Are  appropriate  procedures 
specified  for  applying  the 
qualification  tests  to  BIFV 
gunner  candidates? 


3. 5. 1.3  Are  appropriate  standards  for 
qualification  as  BIFV  gunner 
specified  for  the  candidate 
qualification  tests? 

3.5.2  Does  the  system  produce  appro¬ 
priate  tests  for  assessing 
requirements  for  tailoring  the 
instruction  to  accepted  BIFV 
gunner  -learners  ? 

3.5.2. 1  Do  the  tailoring  tests  provide 
adequate  pre-training  measures 
of  accepted  BIFV  gunner- 
learners'  abilities? 


3. 5. 2. 2  Are  appropriate  procedures  spe¬ 
cified  for  applying  the  tailor¬ 
ing  tests  to  accepted  BIFV 
gunner-learners? 


3. 5. 2. 3  Are  appropriate  standards  for 
tailoring  instruction  specified 
for  the  BIFV  gunnery  tailoring 
tests? 

3.5.3  Does  the  system  produce  appro¬ 
priate  tests  for  measuring  what 
gunner-learners  have  achieved 
as  a  result  of  BIFV  gunnery 
training? 


°  BIFV  gunner-learner  tests  prepared 

°  BIFV  gunnery  testing  applications/ 
requirements  for  which  tests  are 
prepared 

0  BIFV  gunner  behaviors  for  which  the 
tests  are  prepared 

0  Relevance  of  the  tests 

°  Candidate  gunner-learner  qualifica¬ 
tions  for  which  tests  are  prepared 

°  Tests  prepared  for  assessment 
of  those  qualifications 

°  Domains  of  learning  addressed 
by  the  tests 

°  Actions  elicited  by  the  tests 

°  Objects  intended  to  result  from  the 
actions  elicited  by  the  tests 

°  Circumstanc  -s  specified  for  the 
tests 

°  Tools  and  equipment  specified  for 
the  tests 

0  Constraints  specified  for  the  tests 

0  Other  aspects  of  applications  pro¬ 
cedures  specified  for  the  tests 

°  Specific  standards  specified  for 
the  qualification  tests 


°  BIFV  gunnery  abilities  for  which 
tailoring  tests  are  prepared 

°  Tests  prepared  for  assessment  of 
needs  for  tailoring  instruction  to 
fit  gunner-learners’  abilities 

°  Domains  of  learning  addressed  by 
the  tests 

°  Actions  elicited  by  the  tests 

0  Objects  intended  to  result  from  the 
actions  elicited  by  the  tests 

9  Circumstances  specified  for  the  tests 

°  Tools  and  equipment  specified  for 
the  tests 

0  Constraints  specified  for  the  tests 

0  Other  aspects  of  applications  pro¬ 
cedures  specified  for  the  tests 

°  Specific  standards  specified  for 
the  tailoring  teats 


°  BIFV  gunnery  abilities  for  which 
achievement  tests  are  prepared 
*  Tests  prepared  for  assessment  of 
BIFV  gunner-learners'  achievements 
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Subset  No.  13:  (Cont'd) 


3. $.3.1  Do  the  achievement  tests  provide 
adequate  poet-training  measures 
of  BIFV  gunner-learners' 
abilities? 


3.S.3.2  Are  appropriate  procedures  spe¬ 
cified  for  applying  the  achieve¬ 
ment  tests  to  BIFV  gunner- 
learners? 


3. 5. 3. 3  Are  appropriate  standards  for 

BIFV  gunner-learner  achievement 
specified  for  the  achievement 
tests? 


°  Domains  of  learning  addressed  by 
the  tests 

°  Actions  slidted  by  the  tests 
°  Objects  intended  to  result  from  the 
actions  slidted  by  the  tests 

°  Circumstances  apedfled  for  the  tests 
°  Tools  and  equipment  apedfled  for 

the  tests 

°  Constraints  spedfled  for  the  tests 
°  Other  aspects  of  applications  pro¬ 
cedures  spedfled  for  the  testa 

°  Spedflc  standards  spedfled  for 
the  achievement  tests 
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Subset 


3.6 


?.6.1 


3.6. 1.1 


3.6. 1.2 


3.6. 1.3 


3.6.2 


3.6. 2.1 


No.  14:  'Training  Trial  Data"  v 

-  I,,ueg  Attributes - L 


Does  the  system  produce  all 
neceaaary  and  appropriate  train¬ 
ing  trial  data,  including  in 
particular  data  on  BIFV  gunner- 
learner  teating? 


Does  the  ayatem  produce  suffi¬ 
cient  data  to  provide  justifi¬ 
cation  for  specific  features  of 
the  BIFV  gunnery  instructional 
design? 

Does  the  system  produce  appro¬ 
priate  entry  test  data  derived 
from  representative  candidates 
for  BIFV  gunnery  training? 


Does  the  system  produce  appro¬ 
priate  post-test  data  derived 
from  representative  current 
BIFV  gunners? 


Does  the  system  produce  appro¬ 
priate  test  item  error  analyses 
derived  from  tryout  gunner- 
learners? 


Does  the  system  produce  suffi¬ 
cient  data  to  demonstrate  the 
validity  and  utility  of  BIFV 
gunnery  training? 


D°*s  the  system  produce  appro¬ 
priate  test  item  error  analyses 
derived  from  field  test  BIFV 
gunner-learner  s  ? 


Training  trials  from  which  data 
are  obtained 

Types  or  categories  of  data  obtained 
The  data  obtained  within  each 
category 

Completeness  of  the  data 
Accuracy  of  the  data 

The  data  produced  to  provide  justifi¬ 
cation  for  specific  features  of  BIFV 
gunnery  instructional  design 
Gunner-learner  testing  conducted  to 
help  supply  those  data 

Representative  candidates  to  whom 
BIFV  gunnery  entry  testing  is 
administered 

Entry  level  tests  administered  to 

the  representatives 

The  entry  test  data  obtained 

Representative  current  BIFV  gunners 
to  whom  post-testing  is  administered 
Post-tests  administered  to  the 
representatives 
The  post-test  data  obtained 

The  tryout  gunner  learners  to  whom 
test  items  are  administered 
The  test  items  sdministered  in  the 
tryouts 

The  test  item  data  obtained 

The  analyses  of  error  conducted  on 

those  data 

The  data  produced  to  demonstrate  the 
validity  and  utility  of  the  BIFV 
gunnery  training 

Gunner-learner  testing  conducted  to 
help  supply  those  data 

The  field  test  BIFV  gunner-learners 
to  whom  teat  items  are  administered 
The  test  items  administered  in  the 
field  tests 

The  test  item  data  obtained 
The  analyses  of  error  conducted 
on  those  data 
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required  for  detecting  targets,  or  tracking,  or  shooting,  etc.  Unless  the 
system  can  accurately  define  exactly  what  behaviors  gunners  must  be  able  to 
manifest  to  accomplish  effective  gunnery  operations,  the  system  will  not  know 
what  behaviors  should  be  tested .  If  the  training  goals  are  loosely  or 
ambiguously  stated  so  that  the  required  gunner  behaviors  are  not  clear,  it  is 
possible  that  the  entire  structure  built  upon  the  foundation  of  those  goals  will 
be  inappropriate.  That  is,  the  gunnery  performance  objectives,  the  learning 
activities,  the  training  documents,  the  testing  specifications,  and  all  other 
facets  of  the  assembled  instruction  might  be  focused  on  a  set  of  behaviors  that 
are  not  quite  the  ones  that  BIFV  gunners  really  need  to  perform.  The  people 
paying  for  the  training  design  effort  certainly  would  want  some  assurance,  at 
the  outset,  that  the  system  they  are  funding  will  be  able  to  define  the  correct 
gunner  behaviors. 

But,  how  does  one  determine  whether  an  ability  is  present  before 
any  processes  are  carried  out  and  before  any  products  are  delivered? 
Obviously,  one  examines  the  people,  plans  and  tools  assembled  for  carrying 
out  the  processes  and  producing  the  products,  and  tries  to  determine  whether 
they  have  what  it  takes  to  do  the  job  right.  In  the  case  of  the  particular 
ability  to  state  training  goals  in  behaviorally  referenced  terms,  the  evaluator 
would  examine  the  training  design  plans  and  perhaps  speak  with  the  training 
designers  to  ascertain: 

The  behavioral  terminology  they  intend  to  employ  in  their 
statements  of  BIFV  gunnery  training  goals. 

The  formats  they  intend  to  follow  in  preparing  their  statements 
of  BIFV  gunnery  training  goals. 

These  two  attributes  of  Design  subsystem  potentiality  will  allow  the  evaluators 
to  form  reasonable  judgments  about  the  training  designers'  knowledge  of 
behaviorally  referenced  goals,  awareness  of  the  importance  of  such  goals  for 
all  facets  of  training  development  (including  the  specification  of  gunner-learner 
testing),  provisions  they  are  making  for  producing  such  goals,  etc.  In  short, 
these  two  attributes  provide  the  evaluator  with  something  concrete  with  which 
to  deal  in  his  or  her  efforts  to  assess  the  system's  ability  to  specify  the 
proper  kinds  of  training  goals.  After  examining  those  attributes,  an  evaluator 
might  report  back  something  like  the  following  to  the  keepers  of  the  purse 
strings: 

"Look,  I  examined  the  proposal  your  training  designers  submitted, 
and  I  found  nothing  in  it  to  indicate  any  recognition  of  the  need  to 
produce  behaviorally  referenced  training  goals.  I  spoke  to  the 
designers,  and  asked  them  to  give  me  some  examples  of  the  kinds  of 
goal  statements  they  intended  to  develop.  The  examples  they  came 
up  with  were  absolutely  devoid  of  any  behavioral  terms.  They 
simply  weren't  the  kinds  of  statements  that  can  lead  to  a  proper 
analysis  of  learner  bahavioral  requirements.  I  asked  them  if  they 
couldn't  suggest  a  format  for  their  goal  statements  that  would  help 


make  explicit  the  learners'  intended  behavioral  achievements,  and  it 
was  clear  that  they  didn't  really  understand  what  I  was  seeking.  I 
conclude  that,  as  things  now  stand,  your  system  can't  insure  that 
the  goals  will  be  stated  in  terms  that  imply  the  required  gunner 
behaviors'!  I  suggest  that  you  delay  starting  the  development  of 
this  training  until  you've  either  brought  these  designers  up  to 
speed,  or  until  you've  replaced  them  with  some  people  who 
understand  behaviorally  referenced  training.''* 

As  a  second  example,  consider  the  fifth  issue  in  Subset  No.  6, 
"Does  the  system  accurately  identify  the  physical  skills  that  BIFV  gunners 
must  acquire  to  perform  their  tasks,  so  that  appropriate  tests  of  physical 
skills  may  be  specified  and  developed?"  This  issue  arises  from  a 
characteristics-level  process,  i.e.,  the  analytic  process  used  to  identify  the 
physical  skills  learners  must  acquire  in  order  to  serve  as  BIFV  gunners. 
Clearly,  tests  of  all  required  physical  skills  must  be  specified  and  administered 
as  part  of  the  gunner  certification  process.  If  the  system's  analysis  is 
deficient,  so  that  certain  essential  physical  skills  are  not  identified,  or  so  that 
some  irrelevant  physical  skills  are  mistakenly  identified  as  essential,  the 
testing  specifications  also  will  be  deficient.  As  the  design  work  progresses, 
the  sponsors  certainly  would  wish  to  know  whether  a  thorough  and  accurate 
analysis  of  required  physical  skills  is  underway. 

In  order  to  assess  that  issue,  the  evaluator  would  look  first  at  the 
analytic  methods  the  training  designers  are  using  to  identify  required  physical 
skills.  Are  those  methods  appropriate  to  the  analytic  needs  at  hand?  Are  all 
of  the  data  required  for  application  of  those  methods  being  obtained  and  used 
appropriately?  Is  there  any  fault  to  find  with  the  way  in  which  the  training 
designers  are  applying  those  methods?  As  the  analyses  begin  to  produce 
results,  the  evaluator  would  examine  the  physical  skills  requirements 
identified.  Are  all  of  those  skills  really  pertinent  to  BIFV  gunnery  operations? 
Are  there  any  essential  skills  that  haen't  been  identified  in  this  analysis?  Are 
there  any  skills  that  the  designers  have  labelled  "physical"  that  really  are 
"mental?"  Based  on  these  and  similar  considerations,  the  evaluator  can  form  a 
well  reasoned  judgment  about  the  accuracy  of  the  designers'  approach  to 
identifying  physical  skills  requirements.  That  reasoned  judgment  will  be  based 
on  measures  applied  to  the  two  concrete  attributes,  vii.,  the  analytic  methods 
of  skills  identification  and  the  identified  skills  themselves. 

The  evaluative  issues  thus  lead  to  measurable  attributes,  and  the 
attributes  in  turn  lead  to  measures  of  performance  and  effectiveness.  The 
question  is:  How  does  all  this  "leading"  proceed?  How  does  one  discern  the 

•This  sample  "report"  is  purely  hypothetical,  and  is  intended  only  to  illustrate 
how  examination  of  concrete  attributes  could  provide  answers  to  the  question 
posed  in  a  particular  evaluative  issue.  In  no  way  should  the  reader  infer 
that  the  authors  of  this  report  expect  that  the  BIFV  Task  Force  will  produce 
poorly  stated  goals  for  gunnery  training.  The  authors  are  much  impressed 
with  the  experience,  professionalism,  and  dedication  of  the  Task  Force 
members  they  have  met,  and  have  no  doubt  of  the  high  quality  of  their  work. 
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attributes  in  the  issues,  and  how  does  one  derive  the  measures  from  the 
attributes? 

No  precise  or  even  preliminary  guidelines,  algorithms  or  heuristics 
for  extracting  attributes  from  issues  have  thus  far  been  developed.  A  major 
purpose  of  this  trial  application  of  system  performance  taxonomization  to 
evaluating  BIFV  Gunnery  testing  specifications  is  to  generate  an  experiential 
base  from  which  such  guidelines  can  begin  to  be  derived.  But  in  very  general 
terms,  the  analyst's  thought  process  in  searching  for  the  attributes  of  a 
particular  issue  relevant  to  testing  specification  can  be  sketched  this  way: 

"Here  is  an  issue  that  bears  on  how  well  the  BIFV  Gunnery  Design 
Subsystem  has  (or  can)  define  gunner  testing  specifications.  I'm 
supposed  to  suggest  a  way  of  answering  the  question  posed  in  this 
issue.  First,  1  have  to  ask  myself:  which  of  the  subsystem's 
people,  resources/equipment,  and  procedures  are  involved  in  this 
issue?  Next,  what  qualities  or  characteristics  do  those  involved 
people,  resources /equipment,  and  procedures  have  to  possess,  or 
produce  in  their  work,  if  this  issue  is  to  be  resolved  affirmatively? 
Finally,  what  concrete,  observable  factors  associated  with  those 
people,  resources/equipment,  procedures,  or  their  work  can  I 
examine  to  determine  whether  those  qualities  or  characteristics  are 
present? " 

The  concrete,  observable  factors  uncovered  by  the  analyst  are  the 
attributes  associated  with  the  issue  in  question.  The  authors  applied  the 
mental  process  outlined  above  to  each  of  the  119  performance-related  issues 
deemed  relevant  to  evaluation  of  BIFV  gunner  testing  specifications.  The 
outcome  is  the  set  of  attributes  presented  along  with  the  issues  in  Figure  13. 

It  is  the  authors'  considered  opinion  that  the  attribute'  set 
documented  in  Figure  13  is  a  face-valid  and  comprehensive  basis  for  evaluating 
the  adequacy  of  the  gunner-learner  testing  specifications.  But  that  is  only 
opinion,  not  proof.  It  is  hoped  that  the  reviewers  of  this  report,  including 
representatives  of  the  BIFV  Task  Force  and  other  specialists  in  training 
development  and  evaluation,  will  assess  and  critique  the  attribute  set  so  that  a 
much  broader  base  of  expertise  and  insight  will  be  applied  to  the  continuing 
development  of  APM  procedures. 

If  the  reader  will  accept  the  attribute  set  given  in  Figure  13  as 
being  at  least  illustrative  of  the  bases  from  which  the  evaluative  issues  can  be 
addressed,  attention  can  now  turn  to  the  derivation  of  measures  from 
attributes.  A  measure,  in  most  general  terms,  is  a  judgment  or  appraisal 
about  the  thing-being-measured.  Measures  applied  to  system  attributes 
derived  from  various  taxa  of  performance  thus  are  judgments  or  appraisals  of 
performance.  Each  such  measure  contributes  some  small  bit  of  wisdom  or 
insight  about  the  total  performance  of  the  system-being-measured.  Every 
measure  "looks"  at  its  attribute  in  its  own  unique  way,  and  "weighs"  the 
attribute  on  its  own  unique  scales.  Collectively,  the  measures  are  intended  to 
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determine  whether  the  attribute  is  "good  enough"  to  meet  the  demands  of  the 
performance  requirement  from  which  it  derives.  Individually,  each  measure 
focuses  on  some  particular  aspect  or  dimension  of  "goodness." 

Consider  this  example:  the  BIFV  Gunnery  Training  Design 
subsystem  must  have  the  ability  to  insure  that  all  BIFV  gunnery  performance 
objectives  will  be  identified  so  that  they  may  be  addressed  in  gunner  testing 
specifications.  One  attribute  of  the  system's  ability  to  do  that  is  the  scope  of 
information  available  to  it  concerning  the  BIFV  gunners'  tasks.  Gunnery 
performance  objectives  must  be  derived  from  the  gunners'  tasks.  If  the 
system's  knowledge  of  the  gunners'  tasks  is  not  "good  enough,"  it  will  not 
have  the  ability  to  identify  gunnery  performance  objectives  accurately. 

How  does  one  measure  whether  a  set  of  knowledge  is  "good  enough." 
First,  one  specifies  or  states  exactly  what  is  known,  i.e.,  the  contents  of  the 
set  of  knowledge.  Next,  one  identifies  missing  elements  in  the  set  of 
knowledge,  i.e.,  facts  or  other  information  that  should  be  known,  but  aren't. 
Finally,  one  identifies  inaccuracies  in  the  set  of  knowledge,  i.e.,  supposed 
"facts"  and  other  information  that  are  not  true.  This  is  a  roundabout  way  of 
saying  simply  that  a  person's  knowledge  of  a  given  subject  might  be  "bad"  in 
any  of  three  ways: 

the  person  might  know  nothing  about  the  subject;  or 
might  know  some  things  but  not  others;  or 
might  "know"  some  things  wrong. 

A  combination  of  the  last  two  is  also  possible. 

In  the  specific  context  of  the  Design  Subsystem's  knowledge  of  the 
BIFV  Gunner  Tasks,  the  evaluator  would  need  to  determine,  first,  whether  the 
subsystem  knows  anything  about  any  of  the  tasks;  second,  whether  some  tasks 
are  unknown  to  the  subsystem;  and,  third,  whether  some  of  the  "known"  tasks 
are  in  fact  not  tasks  required  of  BIFV  gunners.  Each  of  those  determinations 
provides  a  separate  judgment  or  appraisal  of  the  goodness  of  the  scope  of 
information  the  Design  Subsystem  proposes  to  use  as  the  basis  for  establishing 
BIFV  gunnery  performance  objectives.  That  is,  each  is  a  separate  measure  of 
that  one  attribute  of  the  system's  potential  for  establishing  objectives.  It  is  to 
the  credit  of  the  BIFV  Task  Force  that  they  have  assembled  a  very 
comprehensive  task  data  base  which,  in  fact,  helps  to  meet  the  Potentiality 
aspects  analysis  of  this  exercise. 

Consider  another  example  in  the  context  of  this  same  performance 
potentiality.  Along  with  having  good  information  about  gunnery  tasks,  the 
Design  Subsystem  must  have  people  who  are  qualified  to  do  the  job  of 
establishing  performance  objectives.  The  staff's  experience  and  qualifications 
thus  constitute  another  attribute  of  the  system's  ability  to  produce  well 
founded  objectives.  If  their  qualifications  aren't  "good  enough,"  the  system's 
ability  to  establish  objectives  also  won't  be  good  enough. 
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How  does  one  measure  the  "goodness"  of  an  individual's  or  team's 
qualifications  for  doing  a  particular  job?  When  the  job  is  relatively  simple  and 
of  short  duration,  a  candidate’s  qualifications  sometimes  can  be  measured  by 
requesting  him  or  her  to  actually  do  the  work  in  question,  on  a  sample  basis. 
A  typing  test,  for  example,  often  is  administered  to  candidates  for  a 
secretarial  position.  Then,  actual  job  performance  measures  (e.g.,  words 
typed  per  minute,  percentage  of  errors,  etc.)  can  be  applied  to  the  sample 
work  and  used  as  job  qualification  measures.  However,  when  the  work  is 
complex,  cerebral  and  non-routine,  this  approach  may  not  be  practical.  In 
such  cases,  clinical  judgments  usually  are  employed  to  appraise  the  candidate's 
qualifications.  Subject-matter  experts  review  resumes  of  the  available 
candidates'  experience  and  training  relevant  to  the  job  to  be  filled,  interview 
personal  references  named  by  the  candidates,  perhaps  examine  the  products 
delivered  by  the  candidates  during  previous  assignments  similar  to  the  job  to 
be  filled,  and  maybe  interview  the  candidates  themselves  to  obtain  greater 
insight  concerning  the  skills  and  knowledge  they  could  bring  to  the  job.  Each 
expert  reviewer  then  produces  an  independent  rating  of  each  candidate,  based 
on  the  reviewer's  perception  of  the  candidate's  strengths  and  weaknesses. 
Such  ratings  often  are  placed  on  an  interval  or  ordinal  scale.  Then,  the 
expert  reviewers  meet  as  a  group,  report  and  explain  their  individual  ratings, 
deliberate  on  and  debate  the  issues  raised  concerning  each  candidate,  and  form 
a  consensus  rating  of  the  qualifications  of  each. 

This  clinical  approach  can  be  applied  to  measure  the  Training  Design 
staff's  qualifications  for  establishing  gunnery  performance  objectives.  The 
appropriate  measures  would  include  the  consensus  rating  of  each  individual 
proposed  for  assignment  to  the  task  of  establishing  objectives,  and  an  overall 
consensus  rating  of  the  total  proposed  staff  as  a  team  for  handling  that  task. 
Both  types  of  consensus  ratings  should  include  or  be  augmented  with  detailed 
explanations  oi  all  identified  personnel  deficiencies  bearing  on  the  ability  to 
establish  objectives. 

Referring  next  to  Block  7  of  the  overall  APM  (Figure  1),  it  is 
relatively  easy  to  suggest  measures  for  any  given  attribute.  Unfortunately,  it 
is  quite  something  else  to  devise  a  general  procedure  for  doing  so  in  all  cases. 
Within  the  scope  of  this  current  study,  measures  have  been  suggested  for  all 
of  the  f  5sues  and  attributes  of  four  of  the  taxonomic  subsets  of  Figure  13. 
These  measures  are  arrayed  in  Figures  14  through  17.  The  authors  believe 
that  each  suggested  measure  has  something  of  value  to  say  about  the 
"goodness"  or  "badness"  of  the  attribute  to  which  it  relates.  The  authors  also 
believe  that  the  measures  suggested  for  each  given  attribute  collectively  say 
everything  that  is  pertinent  to  the  "goodness"  or  "badness"  of  that  attribute. 
Obviously,  both  of  these  beliefs  need  to  be  subjected  to  the  test  of  a  careful, 
critical  review  by  the  readers  of  this  report.  It  is  quite  possible,  and 
probably  very  likely,  that  the  measures  hierarchies  shown  in  Figures  14 
through  17  will  undergo  significant  revision  after  readers'  comments  are 
received.  Subsequent  to  such  revision,  the  measures  hierarchies  will  provide 
a  base  from  which  we  can  begin  to  extract  measures  specification  procedures. 
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Hierarchy  Number  1:  BIFV  gunnery  design  subsystem  potential  for 
establishing  performance  objectives 


(Note:  IsBue  numbers  refer  to  the  taxa  in  Figure  7  from  which  they  sre  derived.) 

_ Issues  _ 

1.2  Ability  to  Insure  that  sll  BIFV  gunnery 

performance  objectives  will  be  identi¬ 
fied  so  that  they  may  be  addressed  in 
gunner  testing  specifications 


Attributes 

Scope  of  Information 
available  concerning 
BIFV  gunners'  tasks 


Staff  experience  and 
qualifications  in 
establishing  perfor¬ 
mance  objectives 


Planned  approach  to 
establishing  BIFV 
gunnery  performance 
objectives 


Resources  allocated 
to  establishing  BIFV 
gunnery  prrformanc. >■ 
objectives 


Measures 

A.  BIFV  gunners'  required  tasks  that  are  known  (listing  and  description 
of  each  such  task) 

B.  BIFV  gunners'  required  tasks  that  are  unknown  to  the  Design  Sub¬ 
system  (listing  of  each) 

C.  Tasks  that  actually  are  not  required  of  BIFV  gunners,  but  which  mis¬ 
takenly  are  classified  as  required  by  the  Design  Subsystem  (listing 
and  description) 

D.  Reviewers'  cumulative  ratings  of  each  Design  staff  member's  training, 
experience  and  previous  performance  in  establishing  performance 
objectives  for  training  (using  a  specified  point  allocation  system 

for  training,  experience,  etc.) 

E.  Reviewers'  overall  assessment  of  total  Design  staffs  qualifications 
(based  on  specified  method  of  accumulating  individual  members' 
ratings) 

F.  Essential  steps  missing  from  the  plans  (listing  and  description  of 
each  such  step) 

G.  Inessential  steps  included  in  the  plans  (listing  and  description  of 
each) 

H.  Inconsistencies/deficiencies  in  the  sequencing  of  steps  in  the  plans 

I.  Reviewers'  cumulative  ratings  of  the  soundness /workability  of  the 
plans  (using  a  specified  point  allocation  system) 

J.  Percentage  of  required  person-hours  proposed  for  allocation  to 
establishing  objectives 

K.  Percentage  of  required  materials /goods  proposed  for  allocation  to 
establishing  objectives 

L.  Percentage  of  required  support  services  proposed  for  allocation  to 
establishing  objectives 

M.  Ratio  of  dollar  value  of  total  resources  proposed  for  allocation  to 
establishing  objectives  to  dollar  value  of  total  resources 
actually  required 


Elements  or  factors 
of  gunner  abilities 
intended  to  be 
stated 


Ability  to  insure  that  all  things  BIFV 
gunners  will  be  able  to  do  after  train¬ 
ing  will  be  stated  so  that  tests  of  all 
required  abilities  may  be  specified 

Essential  elements  of  abilities  intended  to  be  included  in  the  state¬ 
ments  of  BIFV  gunner  abilities  (Bating  and  description  of  each  such 
type  of  element) 

Inessential  elements  intended  to  be  included  In  the  statements  of 
BIFV  gunner  abilities  (listing  and  description  of  each  type) 

Essential  elements  of  abilities  not  intended  to  be  included  in  the 
statements  of  BIFV  gunner  abilities  (listing  and  description  of 
each  type 


Figure  14.  Sample  Measures  Hierarchy  Number  1 
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Figure  14.  (Continued) 


1.2. 1.3 


Elements  or  factors 
of  BIFV  gunner  per¬ 
formance  standards 
intended  to  be 
defined 

Plans  for  defining 
the  standards  of 
BIFV  gunnery 
performance 


1.2 


Elements  of  the 
basis  for  assess¬ 
ment  intended  to  he 
provided 


Plans  for  providing 
a  basis  for  objec¬ 
tively  assessing 
BIFV  gunners’ 
performance 


1.2 


Types  of  BIFV  gun¬ 
nery  pretest  items 
intended  to  be 
developed 

Plans  for  develop¬ 
ing  BIFV  gunnery 
pretest  items 


Ability  to  insure  that  definitions  of 
the  appropriate  standards  of  accept¬ 
able  performance  of  the  gunner  actions 
will  be  included  in  the  statements  of 
intended  gunner  abilities,  so  that  the 
specified  tests  may  incorporate  those 
standards 


A.  Relevant  elements  or  factors  of  performance  standards  not  intended 
to  be  defined 

B.  Irrelevant  elements  or  factors  of  performance  standards  intended 
to  be  defined 


C.  Essential  steps  missing  from  the  plans  for  defining  performance 
standards 

D.  Inessential/inappropriate  steps  included  in  the  plans 

E.  Inconsistencies/deficiencies  in  the  sequencing  of  steps  in  the  plans 

F.  Deficiencies  in  the  staff's  abilities  to  carry  out  these  particular 
plans 

G.  Deficiencies  in  the  resources  allocated  to  implementing  these  par¬ 
ticular  plans 

.2  Ability  to  insure  that  the  identified 

BIFV  gunnery  performance  ojbectives 
will  be  applied  to  provide  a  basis  for 
objectively  assessing  gunners' 

_ performance 

A.  Essential  elements  or  segments  of  the  basis  for  objective  assess¬ 
ment  of  BIFV  gunner  performance  intended  to  be  provided  (listing 
and  description  of  each) 

B.  Inessential  elements  or  segments  of  the  basis  intended  to  be  pro¬ 
vided  (listing  and  description) 

C.  Essential  elements  or  segments  of  the  basis  not  intended  to  be 
provided  (listing  and  description) 

D.  Essential  steps  missing  from  the  plans  for  providing  a  basis  for 
objective  assessment 

E.  Inessential/inappropriate  steps  included  in  the  plans 

F.  Inconsistencies/deficiencies  in  the  sequencing  of  steps  in  the  plans 

G.  Deficiencies  in  the  staff’s  abilities  to  carry  out  these  particular 
plans 

H.  Deficiencies  in  the  resources  allocated  to  implementing  these  par¬ 
ticular  plans 


.2.1  Ability  to  insure  that  appropriate  pre- 
test  items  addressing  the  performance 
objectives  will  be  developed  and  in- 
cluded  in  the  testing  specifications 

A.  Relevant  types  of  pretest  items  not  intended  to  be  developed  (listing 
and  description) 

B.  Irrelevant/inappropriate  types  of  pretest  items  intended  to  be 
developed  (listing  and  description) 

C.  Essential  steps  missing  from  the  plans  for  developing  pretest  items 

D.  Inessential/inappropriate  steps  included  in  the  plans 

E.  Inconsistencies/deficiencies  in  the  sequencing  of  steps  in  the  plans 

F.  Deficiencies  in  the  staff's  abilities  to  carry  out  these  particular 
plans 

G.  Deficiencies  in  the  resources  allocated  to  implementing  these  par¬ 
ticular  plans 


Figure  14.  (Continued) 


(Note:  Issue  numbers  refer  to  the  tax*  in  Figure  7  from  which  they  are  derived.) 


2.3 


Attributes 

Task  analytic 
methods  applied  to 
the  BIFV  gunner 

tasks 


The  scope  of  the 
BIFV  gunner  task 
analyses  conducted 


Completeness  of  the 
BIFV  gunner  task 
analyses  conducted 


Accuracy  of  the 
BIFV  gunner  task 
analyses  conducted 


_ _  Issues 

Conduct  of  analysis  of  BIFV  gunner 
tasks,  so  that  appropriate  tests  of 
_ those  tasks  may  be  specified _ 

Measures 

A.  Deviations  between  the  methods  actually  applied  and  the  methods  spe¬ 
cified  in  the  plans  for  analyses  (listing  and  description  of  each 

such  deviation) 

B.  Aspects  or  elements  of  the  methods  applied  that  are  irrelevant  to  or 
inconsistent  with  the  analytic  purposes  (listing  and  description 

of  each) 

C.  Aspects  or  elements  missing  from  the  methods  applied  that  are  essen¬ 
tial  for  the  analytic  purposes  (listing  and  description  of  each) 

D.  Types  of  data  needed  for  the  methods  that  were  not  obtained  (listing 
and  description  of  each  data  deficiency) 

£.  Factors  supposed  to  be  determined  from  the  task  analyses  that 
actually  were  not  determined 

F.  Factors  irrelevant  to  the  analytic  purposes  that  were  determined 
in  the  task  analyses 

G.  Relevant  BIFV  gunner  tasks  to  which  the  analytic  methods  were  not 
applied 

H.  Factors  missing  from  the  analyses  applied  to  other  relevant  BIFV 
gunner  tasks 

I.  Inaccuracies  in  the  input  data  obtained  for  application  of  the 
analaytic  methods 

J.  Specific  misinterpretations  of  the  analysed  data 

K.  Specific  aspects  or  elements  of  the  analytic  methods  that  were 
misapplied 


Classification  of  BIFV  gunner  tasks 
in  terms  of  the  domains  of  learning 
_ they  involve _  _ 


Methods  employed  to  A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
identify  the  do-  or  inconsistent  with  identification  of  the  domains  of  learning 

mains  of  learning  involved  in  a  task  (listing  and  description) 

of  the  BIFV  gunner  B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
tasks  essential  for  identifying  the  domains  of  learning  (listing  and 

description) 

C.  Input  data  required  for  the  methods  that  are  missing  or  insufficient 
(listing  and  description) 


The  domains  of  D.  Domains  identified  as  involved  in  a  task  that  actually  are  not  rele- 

leamlng  Identified  vant  to  that  task  (listing  and  description  of  each  such  incident) 

lor  each  BIFV  gun-  E.  Domains  not  identified  as  involved  in  a  task  that  actually  are  in¬ 
ner  task  volved  in  that  task  (listing  and  description  of  each  such  incident) 


Figure  15.  Sample  Measures  Hierarchy  Number  2 
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2. 3. 2.1 


Methods  employed 

A.  . 

to  identify  the 

i 

knowledge  require¬ 

1 

ments  of  BIFV 

B.  , 

gunner  tasks 

t 

4 

c.  : 

! 

The  knowledge 

D.  1 

requirements  iden¬ 

tified  for  BIFV 

i 

gunner  tasks 

E.  1 

Identification  of  the  knowledge  that 
BIFV  gunners  must  possess  in  order 
to  perform  their  tasks,  so  that 
appropriate  tests  of  knowledge  may 
_ be  specified  and  developed _ 

Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  knowledge  required  for 
performing  a  task  (listing  and  description) 

Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  identification  of  the  knowledge  required  for  per¬ 
forming  a  task  (listing  and  description) 

Input  data  required  for  the  methods  that  are  missing  or  insuf¬ 
ficient  (listing  and  description) 

Knowledge  elements  identified  as  required  for  a  given  task  that 
actually  are  not  needed  for  performance  of  that  task  (listing  and 
description) 

Knowledge  elements  not  identified  as  required  for  a  given  task  that 
actually  are  needed  for  performance  of  that  task  (listing  and 
description) 


2. 3. 2. 2  Identification  of  the  mental  skills 

that  BIFV  gunners  must  possess  in 
order  to  perform  their  tasks,  so  that 
appropriate  tests  of  mental  skills 
_ may  be  specified  and  developed 


Methods  employed  A. 

to  identify  the 
mental  skill 

requirements  of  B. 

BIFV  gunner 

tasks 

C. 


The  mental  skill  D. 

requirements  iden¬ 
tified  for  BIFV  E. 

gunner  tasks 


Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  mental  skills  required 
for  performing  a  task  (listing  and  description) 

Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  identification  of  the  mental  skills  required  for 
performing  a  task  (listing  and  description) 

Input  data  required  for  the  methods  that  are  missing  or  insuffi¬ 
cient  (listing  and  description) 

Mental  skills  identified  as  required  for  a  given  task  that  actually 
are  not  needed  for  performance  of  that  task  (listing  and  description) 
Mental  skills  not  identified  as  required  for  a  given  task  that  actu¬ 
ally  are  needed  for  performance  of  that  task  (listing  and  descrip¬ 
tion 


2. 3. 2. 3  Identification  ol  the  physical  skills 
that  BIFV  gunners  roust  possess  in 
order  to  perform  their  tasks,  so  that 
appropriate  tests  of  physical  skills 
_ may  be  specified  and  developed 


Methods  employed  to 
identify  the  physi¬ 
cal  skill  require¬ 
ments  of  BIFV 
gunner  tasks 


A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  physical  skills 

required  for  performing  a  task  (listing  and  description) 

B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  Identification  of  the  physical  skills  required  for 
performing  a  task  (listing  and  description) 

C.  Input  data  required  for  the  methods  that  are  missing  or  insuffi¬ 
cient  (listing  and  description) 


The  physical  skill 
requirements  iden¬ 
tified  for  BIFV 
gunner  tasks 


D,  Physical  skills  identified  as  required  for  a  given  task  that  actually 
are  not  needed  for  performance  of  that  task  (listing  and  description) 

E.  Physical  skills  not  identified  as  required  for  a  given  task  that 
actually  are  needed  for  performance  of  that  task  (listing  and 
description) 


Figure  1$.  (Continued) 
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2. 3. 2.4 


Methods  employed  to 

A.  , 

identify  the  attitu¬ 

i 

dinal  requirements 

1 

of  BIFV  gunner 

B.  . 

tasks 

i 

i 

1 

c.  : 

The  attitudinal 

D.  , 

requirements  iden¬ 

tified  for  BIFV 

i 

gunner  tasks 

E.  . 

Identification  of  the  attitudes  that 
BIFV  gunners  must  possess  in  order 
to  perform  their  tasks,  so  that  appro¬ 
priate  tests  of  attitudes  may  be 
_ specified  and  developed _ 


Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  attitudes  required  for 
performing  a  task  (listing  and  description) 

Aspects  or  elements  missing  from  the  methods  employed  that  are 
essentia]  for  identification  of  the  attitudes  required  for  perform¬ 
ing  a  task  (listing  and  description) 

Input  data  required  for  the  methods  that  are  missing  or  insuffi¬ 
cient  (listing  and  description 

Attitudinal  elements  identified  as  required  for  a  given  task  that 
actually  are  not  needed  for  performance  of  that  task  (listing  and 
description) 

Attitudinal  elements  not  identified  as  required  for  a  given  task  that 
actually  are  needed  for  performance  of  that  task  (listing  and 
description) 


2.3.3  Identification  of  the  conditions  for 

_ learning  the  BIFV  gunner  tasks 


Methods  employed  to 
identify  the  condi¬ 
tions  for  learning 
the  BIFV  gunner 
tasks 


Types  of  conditions 
for  learning  that 
are  identified  for 
each  BIFV  gunner 
task 


A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  conditions  for  learning 

a  task  (listing  and  description  of  each  such  aspect) 

B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  identifying  the  conditions  for  learning  (listing  and 
description 

C.  Input  data  required  for  the  methods  that  are  missing  or  insuffi¬ 
cient  (listing  and  description) 

D.  Types  of  conditions  for  learning  identified  as  associated  with  a 
task  that  actually  are  not  relevant  to  that  task  (listing  and 
description) 

E.  Types  of  conditions  for  learning  not  identified  as  associated  with 
a  task  that  actually  are  relevant  to  that  task  (listing  and 
description) 


2.3. 3.1  Identification  of  the  prerequisite 

knowledge  that  BIFV  gunners  should 
have  in  order  to  learn  their  tasks 


Methods  employed 
to  identify  the 
prerequisite 
knowledge  for 
BIFV  gunner 
tasks 


Knowledge  items 
identified  as 
prerequisites  for 
learning  the 
BIFV  gunner 
tasks 


A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  knowledge  prerequisites 
required  for  learning  a  task 

B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  identification  of  the  knowledge  prerequisites 
required  for  learning  a  task 

C.  Input  data  required  for  the  methods  that  are  missing  or  insufficient 

D.  Knowledge  items  identified  as  prerequisites  for  learning  a  given 
task  that  actually  are  not  needed  or  helpful  for  learning  the 
task  (listing  and  description) 

E.  Knowledge  items  not  identified  as  prerequisites  for  learning  a 
given  task  that  actually  are  essential  for  learning  the  task 

F.  Knowledge  items  not  identified  as  prerequisites  for  learning  a 
given  task  that  actually  would  facilitate  learning  the  task, 
although  not  essential  for  learning 


Figure  1%.  (Continued) 
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A 
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2. 3. 3. 2  Identification  of  the  prerequisite 

mental  skills  that  BIFV  gunners 
should  have  in  order  to  learn 
their  tasks 


Methods  employed 
to  identify  the 
prerequisite 
mental  skills  for 
BIFV  gunner 
tasks 


Specific  mental 
skills  identified 
as  prerequisites 
for  learning  the 
BIFV  gunner 
tasks 


A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  mental  skills  pre¬ 
requisites  for  learning  a  task 

B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  identification  of  the  mental  skills  prerequisites 

for  learning  a  task 

C.  Input  data  required  for  the  methods  that  are  missing  or  insufficient 

D.  Specific  mental  skills  identified  as  prerequisites  for  learning  a 
given  task  that  actually  are  not  needed  or  helpful  for  learning 
the  task 

E.  Specific  mental  skills  not  identified  as  prerequisites  for  learning 
a  given  task  that  actually  are  essential  for  learning  the  task 

F.  Specific  mental  skills  not  identified  as  prerequisites  for  learning 
a  given  task  that  actually  would  facilitate  learning  the  task, 
although  not  essential  for  learning 


2. 3. 3. 3  Identification  of  the  prerequisite 

physical  skills  that  BIFV  gunners 
should  have  in  order  to  learn 
their  tasks 


Methods  employed 
to  identify  the 
prerequisite 
physical  skills 
for  BlfV 
gunner  tasks 


•Specific  physical 
skills  identified 
as  prerequisites 
for  learning  the 
BIFV  gunner 
tasks 


A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  physical  skills  pre¬ 
requisites  for  learning  a  task 

B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  identification  of  the  physical  skills  prerequisites 
for  learning  a  task 

C.  Input  data  required  for  the  methods  that  are  missing  or  insufficient 

D.  Specific  physical  skills  identified  as  prerequisites  for  learning  a 
given  task  that  actually  are  not  needed  or  helpful  for  learning 
the  task 

E.  Specific  physical  skills  not  identified  as  prerequisites  for  learning 
a  given  task  that  actually  are  essential  for  learning  the  task 

F.  Specific  physical  skills  not  identified  as  prerequisites  for  learning 
a  given  task  that  actually  facilitate  learning  the  task,  although 
not  essential  for  learning 


Methods  employed 
to  identify  the 
prerequisite  atti¬ 
tudes  for  BIFV 
gunner  tasks 


Specific  ati.tudes 
identified  as  pre¬ 
requisites  for 
learning  the 
BIFV  gunner 
tasks 


2. 3. 3. 4  Identification  of  the  prerequisite 

attitudes  that  BIFV  gunners  should 
have  in  order  to  learn 
_ their  tasks 

A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  attitudinal  prerequi¬ 
sites  for  learning  a  task 

B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
essential  for  identification  of  the  attitudinal  prerequisites  for 
learning  a  task 

C.  Input  data  required  for  the  methods  that  are  missing  or  insufficient 

D.  Specific  attitudes  identified  as  prerequisites  for  learning  a  given 
task  that  actually  are  not  needed  or  helpful  for  learning  the  task 

E.  Sp  ecific  attitudes  not  identified  as  prerequisites  for  learning  a 
given  task  that  actually  are  essential  for  learning  the  task 

F.  Specific  attitudes  not  identified  as  prerequisites  for  learning  a 
given  task  that  actually  facilitate  learning  the  task,  although 
not  essential  for  learning 


Figure  15.  (Concluded) 


90d 


Hierarchy  Number  3:  BIFV  gunnery  design  subsystem  product 
consisting  of  stated  performance  objectives 

(Note:  Issue  numbers  refer  to  the  taxa  in  Figure  7  from  which  they  are  derived.) 

Issues 

3.2  Production  of  accurate  and  complete 

statements  of  BIFV  gunner  performance 
objectives,  so  that  tests  of  gunners' 

_ qualifications  may  be  specified _ 


Attributes 


Measures 


The  BIFV  gunner 
performance 
objectives  that 
are  stated 


A.  Statements  that  define  performance  objectives  that  actually  are  not 
needed  for  BIFV  gunnery  (listing  and  description  of  each  such 
statement) 

B.  Performance  objectives  that  actually  are  needed  for  BIFV  gunnery 
that  are  not  addressed  in  any  of  the  statements  produced  (listing 
and  description  of  each  such  objective) 


The  format  of  the  C. 

statements  of 
BIFV  gunner 

performance  D. 

objectives 


The  completeness  E. 

of  the  statements 
of  BIFV  gunner 
performance 
objectives 

The  accuracy  of  F. 

the  statements  of 
BIFV  gunner 

performance  G. 

objectives 


3.2.1 


The  BIFV  gunner- 
learner  achieve¬ 

A. 

ments  that  are 
defined 

B. 

C. 

D. 

The  factors  in¬ 
cluded  in  the 
definitions  of 

E. 

BIFV  gunner- 
learner  achieve¬ 

F. 

ments 

Essential  elements  of  performance  objective  statements  not  included 
in  the  format  employed  for  stating  BIFV  performance  objectives 
(listing  and  description  of  each  missing  element) 

Inessential/inappropriate  elements  in  the  format  employed  for  stat¬ 
ing  BIFV  performance  objectives  (listing  and  description  of  each 
such  element) 

Elements  missing  from  the  statements  of  particular  BIFV  perfor¬ 
mance  objectives  (tabulation  of  missing  elements  as  a  function  of 
the  stated  objectives) 


BIFV  performance  objective  statements  that  contain  erroneous  or 
inaccurate  elements  (tabulation  of  inaccurate  elements  as  a 
function  of  the  stated  objectives) 

Description  of  the  nature  of  the  inaccuracies,  for  each  stated 
objective 


Production  of  exact  definitions  of 
the  BIFV  gunner-learners'  expected 
achievements  as  a  result  of 
_  the  training _ 

Achievements  defined  that  actually  are  not  necessarily  expected  of 
BIFV  gunner-learners  during  training  (listing  and  description  of 
each  such  achievement) 

Achievements  not  defined  that  actually  are  expected  of  BIFV 
gunner-learners  during  training  (listing  and  description  of  each 
such  achievement) 

Inexact  or  ambiguous  elements  in  the  definitions  of  BIFV  gunner- 
learner  achievements  (listing  and  description) 

Inaccurate  elements  in  the  definitions  of  BIFV  gunner-learner 
achievements  (listing  and  description) 

Factors  included  in  the  definitions  that  actually  are  irrelevant 
to  the  description  of  intended  achievements  of  BIFV  gunner- 
learners  (listing  and  description) 

Factors  missing  from  the  definitions  that  actually  are  essential 
for  describing  intended  achievemente  of  BIFV  gunner-learners 
(listing  and  description) 


Figure  16.  Sample  Measures  Hierarchy  Number  3 
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3.2 


Capabilities  spe¬ 
cified  for 
achievement 
by  B1FV 
gunner-learners 


3.2 


The  actions  speci¬ 
fied  for  execution 
by  BIFV  gunner- 
learners 

The  observability 
of  the  specified 
actions 


\2. 


Objects  specified 
as  results  of  the 
actions  to  be  exe 
cuted  by  BIFV 
gunner-learners 


Measurability  of 
the  specified 
objects 


1 . 1  Production  of  correct  specifications 

of  the  capabilities  that  BIFV 
gunner-learners  are  expected 
to  achieve 


A.  Specified  capabilities  that  actually  are  not  relevant  to  the  intended 
achievements  of  BIFV  gunner-learners  (listing  and  description) 

B.  Capabilities  not  specified  that  actually  are  essential  to  the 
intended  achievements  of  BIFV  gunner-learners  (listing  and 
description) 


1.2  Production  of  correct  specifications 

of  the  actions  that  BIFV  gunner- 
learners  are  expected  to  execute  in 
order  to  demonstrate  each  capability 

A.  Specified  actions  that  actually  do  not  necessarily  demonstrate  the 
intended  capabilities  (listing  and  description) 

B.  Actions  not  specified  that  actually  would  demonstrate  the  intended 
capabilities  (listing  and  description) 

C.  Reviewers'  ratings  of  the  inherent  observability  of  the  specified 
actions  (Likert  or  numeric  scale  ratings) 

D.  Requirements  for  imposing  artificial  constraints  on  the  gunner- 
learners'  execution  of  the  actions  to  enhance  observability 
(listing  and  description) 

£.  Reviewers'  ratings  of  the  degree  to  which  observation  will  impede 
or  interfere  with  gunner-learners'  execution  of  the  actions 
(Likert  or  numeric  scale  ratings) 

1 .3  Production  of  correct  specifications 

of  the  objects  that  are  to  result 
from  the  actions  executed  by  the 

BIFV  gunner-learners _ 

A.  Specified  objects  that  actually  do  not  necessarily  result  from  exe¬ 
cution  of  the  actions  of  BIFV  gunner-learners  (listing  and 
description) 

B.  Objects  not  specified  that  actually  would  result  from  execution  of 
the  actions  of  BIFV  gunner-learners  (listing  and  description) 

C.  Specified  attributes/ features  of  the  objects  that  do  not  actually 
indicate  achievement  of  the  intended  gunner-learner  capabilities 
(listing  and  description) 

D.  Non-specified  attributes/features  of  the  objects  that  actually  are 
important  indicators  of  achievement  of  the  intended  gunner-learner 
capabilities  (listing  and  description) 

E.  Potential  methods  of  direct  measurement  of  the  relevant  attributes/ 
features  of  the  specified  objects  (listing  and  description  of 

each  method) 

F.  Potential  methods  of  indirect /inferential  measurement  of  the 
relevant  attributes/ features  of  the  specified  objects  (listing 
and  description  of  each  method) 

G.  Probabilistic  distribution  of  measurement  error  for  each  of  the 
potential  measurement  methods 


Figure  16.  (Continued) 
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3. 2. 1.4  Production  o(  correct  specifications 
of  the  circumstances  under  which 
the  BIFV  gunner-learners  are  to  exe- 
cute  the  actions  to  produce  the  objects 


Circumstances 
specified  for  exe¬ 
cution  of  the 
actions  by  BIFV 
gunner-learners 


A.  Specified  circumstantial  factors/categories  that  actually  do  not 
apply  to  execution  of  the  actions  under  realistic  job  conditions 
(listing  and  description  of  each  such  factor/ category ) 

B.  Non-specified  circumstantial  factors/categories  that  actually  do 
apply  to  execution  of  the  actions  under  realistic  job  conditions, 
and  that  might  affect  the  execution  of  the  actions  (listing  and 
description  of  each) 

C.  Deviations  between  the  values,  ranges,  limits,  etc.,  specified  for 
circumstantial  factors/categories  and  the  values,  ranges,  limits, 
etc.,  that  actually  exist  in  those  factors/categories  under  realistic 
job  conditions  (magnitude  and  direction  of  each  such  deviation) 


3.2. 1.5  Production  or  correct  specifications 
of  the  tools  and  equipment  to  be 
used  by  BIFV  gunner-learners  in 
_ executing  the  actions _ 


Tools  and  equip¬ 
ment  specified  for 
use  in  executing 
the  gunner- 
learners'  actions 


A.  Specified  tools  and  equipment  that  actually  would  not  be  available 
or  employed  under  realistic  job  conditions  (listing  and  description) 

B.  Tools  and  equipment  not  specified  that  actually  would  be  applicable 
and  available  under  realistic  job  conditions  (listing  and 
description) 


_ .. » 

■f/J  * 

m 


V,  1 


3.2. 1.6 


Constraints  speci¬ 

A. 

fied  for  the  exe¬ 

cution  of  the 

gunner-learners1 

B. 

actions 

C. 

Criteria  specified 
as  standards  of 
adequacy  for  the 
objects 


3.2. 1.7 


A. 


Production  of  correct  specifications 
of  the  constraints  to  be  imposed 
upon  the  execution  of  actions 
by  BIFV  gunner-learners _ 

Specified  constrained  behaviors/activities  that  actually  would  not 
be  constrained  under  realistic  job  conditions  (listing  and 
description) 

Behaviors/activities  not  specified  for  constraint  that  actually 
would  be  constrained  under  realistic  job  conditions  (listing  and 
description) 

Deviations  between  the  degree  of  constraints  specified  and  the 
degree  of  constraints  that  actually  would  exist  under  realistic  job 
conditions  (magnitude  and  direction  of  each  such  deviation) 


Production  of  correct  specifications 
of  the  criteria  to  be  used  to  judge 
the  adequacy  of  the  objects  resulting 
from  the  actions  executed  by  the 
_ BIFV  gunner-learners _ 

Deviations  between  the  criteria  specified  and  the  standards  actually 
required  for  adequate  performance  on  the  job  (magnitude  and 
direction  of  each  such  deviation) 


Figure  16.  (Concluded) 
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Hierarchy  Number  4:  BIFV  gunnery  design  subsystem  product 
consisting  of  testa 

Issue  numbers  refer  to  the  taxa  in  Figure  7  from  which  they  are  derived.) 


Issues 


3.5 

Production  of  appropriate  tests  for 
administration  to  BIFV  gunner-learners 

Attributes 

Measures 

The  BIFV  gunner- 
learner  tests  that 
are  prepared 

A.  Prepared  tests  that  address  each  BIFV  gunnery  performance  objec 
tive  (listing  and  description  as  a  function  of  each  performance 
objective) 

B.  Prepared  tests  that  relate  to  no  BIFV  gunnery  performance  objec¬ 
tive  (listing  and  description  of  each  such  test) 

BIFV  gunnery 
testing  applica¬ 
tion/requirements 
for  which  tests 
are  prepared 

C.  Reviewers*  ratings  of  the  utility  of  each  prepared  test  for  each 
testing  application /requirement  (yes/no  rating  of  each  test  for 
each  application) 

BIFV  gunner 
behaviors  for 
which  tests  are 
prepared 

D.  Listing  of  tests  prepared  for  each  behavior  associated  with  each 
performance  objective 

Relevance  of  the 
tests 

E.  Deviations  between  the  behaviors  elicited  by  the  tests  and  the 
behaviors  that  actually  are  to  be  addressed  (listing  and  descrip¬ 
tion  of  each  such  deviation) 

Candidate  gunner- 
learner  qualifi¬ 
cations  for  which 
tests  are  prepared 


The  tests  pre¬ 
pared  for  assess¬ 
ment  of  the 
qualifications 


1  Production  of  appropriate  tests  for 

assessing  the  qualifications  of 
_ candidate  BIFV  gunner-learners 

A.  Relevant  qualifications  for  which  tests  are  prepared  (listing  and 
description  of  each  such  qualification) 

B.  Relevant  qualifications  for  which  no  tests  are  prepared  (listing 
and  description  of  each  such  qualification) 

C.  Irrelevant  "qualifications"  for  which  tests  are  prepared  (listing 
and  description  of  each) 

D.  Elements  of  each  test  that  deviate  from  the  qualifications  actu¬ 
ally  required  of  acceptable  gunner-learners  (listing  and  description 
of  each  such  deviating  element) 

E.  Probability  that  each  test  will  be  passed  by  candidates  who 
actually  do  not  possess  the  qualification  being  tested 

F.  Probability  that  each  test  will  be  failed  by  candidates  who 
actually  do  possess  the  qualification  being  tested 

G.  Cumulative  probability  that  an  actually  unqualified  candidate  will 
pass  sufficient  qualification  tests  to  be  accepted  as  a  gunner- 
learner 

H.  Cumulative  probability  that  an  actually  qualified  candidate  will 
fail  sufficient  qualification  tests  to  be  rejected  ss  a  gunner- 
learner 


Figure  17.  Sample  Measures  Hierarchy  Number  4 
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3 . 5 . 1 . 1  Production  of  qualification  tests  that 
provide  adequate  measures  of  candidate 
BIFV  gunner-learners' 
_ prerequisite  abilities _ 


Domains  of 
learning  addressed 
by  the  tests 


Actions  elicited 
by  the  tests 


Objects  intended 
to  result  from 
the  actions 
elicited  by  the 
tests 


A.  Qualification  tests  that  can  only  be  passed  via  application  of  a 
domain  of  learning  other  than  the  domain  relevant  to  the  qualifi¬ 
cation  supposedly  to  be  tested  (listing  and  description  of  each 
such  test) 

B.  Qualification  tests  that  can  also  be  passed  via  application  of  a 
domain  of  learning  other  than  the  domain  relevant  to  the  qualifi¬ 
cation  supposedly  to  be  tested  (listing  and  description  of  each 
such  test) 

C.  Deviations  between  the  actions  elicited  by  the  tests  and  the  actions 
required  to  demonstrate  each  qualification  (listing  and  descrip¬ 
tion  of  each  such  deviation) 

D.  Probability  that  the  execution  of  the  elicited  action  implies  the 
ability  to  execute  the  action  required  to  demonstrate  the 
qualification 

E.  Probability  that  a  failure  to  execute  the  elicited  action  implies 
the  inability  to  execute  the  action  actually  required  to  demon¬ 
strate  the  qualification 

F.  Deviations  between  the  objects  Intended  to  result  from  the  test 
actions  and  the  objects  intended  to  result  from  the  actually  required 
actions  (listing  and  description  of  each  such  deviation) 

G.  Probability  that  the  production  of  the  test  object  implies  the 
ability  to  produce  the  actually  required  object 

H.  Probability  that  a  failure  to  produce  the  test  object  implies  the 
inability  to  produce  the  actually  required  object 


3.5. 1.2  Production  of  appropriate  procedures 
for  applying  the  qualification  tests  to 
_ BIFV  gunner-learner  candidates 


Circumstances 
specified  for 
the  tests 


Tools  and  equip¬ 
ment  specified 
for  the  tests 


A.  Deviations  between  the  circumstances  specified  for  the  tests  and 
the  circumstances  under  which  the  actual  qualification  abilities 
need  to  be  applied  during  training  (listing,  description,  magni¬ 
tude  and  direction  of  each  such  deviation) 

B.  Probability  that  passing  the  test  under  the  specified  circum¬ 
stances  implies  the  candidate  can  apply  the  qualification  ability 
during  training 

C.  Probability  that  failing  the  test  under  the  specified  circum¬ 
stances  implies  the  candidate  cannot  apply  the  qualification 
ability  during  training 

D.  Deviations  between  the  tools  and  equipment  specified  for  the 
tests  and  the  tools  and  equipment  to  be  used  by  gunner-learners 
to  apply  the  qualification  abilities  during  training  (listing  and 
description  of  each  such  deviation) 

E.  Probability  that  passing  the  test  using  the  specified  tools  and 
equipment  implies  the  candidate  can  apply  the  qualification  ability 
using  the  tools  and  equipment  to  be  provided  during  training 

F.  Probability  that  falling  the  test  using  the  specified  tools  and 
equipment  implies  the  candidate  cannot  apply  the  qualification 
ability  using  the  tools  and  equipment  to  be  provided  during  training 


Figure  17.  (Continued) 


Constraints  speci¬ 
fied  for  the  tests 


3.5. 


■Standards  speci¬ 
fied  for  the 
qualification 
tests 


5.5. 


BIFV  gunnery 
abilities  for 
which  tail;  'dug 
tests  a • e  pre¬ 
pared 

The  tests  prepared 
for  assessment  of 
the  needs  for 
tailoring  instruc¬ 
tion  to  (it 
learners'  abili¬ 
ties 


3.5. 


Domains  of  learn¬ 
ing  addressed  by 
the  tests 


G.  Deviations  between  the  constraints  specified  for  the  tests  and  the 
actual  constraints  governing  the  gunner-learners'  application  of 
the  qualification  abilities  during  training  (listing  and  description 
of  each  such  deviation) 

H.  Probability  that  passing  the  test  under  the  specified  constraints 
implies  that  the  candidate  can  apply  the  qualification  ability 
under  the  constraints  that  will  exist  during  training 

I.  Probability  that  failing  the  test  under  the  specified  constraints 
implies  that  the  candidate  cannot  apply  the  qualification  ability 
under  the  constraints  that  will  exist  during  training 

1 . 3  Production  of  appropriate  qualification 
test  standards  for  accepting  candidate 
BIFV  gunner-learners  into  training 

A.  De  viations  between  the  specified  test  standards  and  the  actual 
standards  of  adequate  performance  of  the  qualification  abilities 
during  training  (listing,  description,  magnitude  and  direction  of 
each  such  deviation 

B.  Probability  that  satisfaction  of  the  test  standards  implies  that 
the  candidate  can  satisfy  the  actual  standards  of  adequate  per¬ 
formance  during  training 

C.  Probability  that  failure  to  satisfy  the  test  standards  implies 
that  the  candidate  cannot  satisfy  the  actual  standards  of 
adequate  performance  during  training 

2  Production  of  appropriate  tests  for 

assessing  requirements  for  tailoring 
the  instruction  to  accepted 
_  BIFV  gunner-learners _ 

A.  Relevant  gunnery  abilities  for  which  tests  are  prepared  (listing  and 
description  of  each  such  ability) 

B.  Relevant  gunnery  abilities  for  which  no  tests  are  prepared  (listing 
and  description  of  each  such  ability) 

C.  Irrelevant  "abilities"  for  which  tests  are  prepared  (listing- and 
description  of  each) 

D.  Elements  of  each  test  that  deviate  from  the  abilities  actually 
required  of  BIFV  gunners  (listing  and  description  of  each  such 
deviating  element) 

E.  Probability  that  each  test  will  be  passed  by  learners  who 
actually  do  not  possess  the  ability  being  tested 

F.  Probability  that  each  test  will  be  failed  by  learners  who 
actually  do  possess  the  ability  being  tested 

G.  Cumulative  probability  lhat  a  learner’s  test  performance  will 
result  in  inappropriate  tailoring  of  instruction  for  that  learner 

l .  1  Production  of  tailoring  tests  that 

provide  adequate  pre-training 
measures  of  accepted  BIFV 
_  gunner-learners'  abilities _ 

A.  Tailoring  tests  that  can  only  be  passed  via  application  of  a 
domain  of  learning  other  than  the  domain  relevant  to  the  ability 
supposedly  to  be  tested  (listing  and  description  of  each  such  test) 

B.  Tailoring  tests  that  can  also  be  passed  via  application  of  a 
domain  of  learning  other  than  the  domain  relevant  to  the  ability 
supposedly  to  be  tested  (listing  and  description  of  each  such  test) 


Figure  17.  (Continued) 
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Actions  elicited 
by  the  tests 


Objects  intended 
to  result  from  the 
actions  elicited 
by  the  tests 


3.5. 


Circumstances 
specified  for 
the  tests 


Tools  and  equip¬ 
ment  specified 
for  the  tests 


Constraints  speci¬ 
fied  for  the  tests 


C.  Deviations  between  the  actions  elicited  by  the  tests  and  the 
actions  required  to  demonstrate  each  tested  ability  (lifting  and 
description  of  each  such  deviation) 

D.  Probability  that  the  execution  of  the  elicited  action  implies  the 
learner  can  execute  the  action  actually  required  to  demonstrate 
the  tested  ability 

£.  Probability  that  a  failure  to  execute  the  elicited  action  implies 
the  learner  cannot  execute  the  action  actually  required  to 
demonstrate  the  tested  ability 

F.  Deviations  between  the  objects  intended  to  result  from  the  test 
actions  and  the  objects  intended  to  result  from  the  actually 
required  actions  (listing  and  description  of  each  such  deviation) 

G.  Probability  that  the  production  of  the  test  object  implies  the 
ability  to  produce  the  actually  required  object 

H.  Probability  that  a  failure  to  produce  the  test  object  implies 
the  inability  to  produce  the  actually  required  object 


2.2  Production  of  appropriate  procedures 
for  applying  the  tailoring  tests 
_ to  BIFV  gunner-learners 

A.  Deviations  between  the  circumstances  specified  for  the  tests  and 
the  circumstances  under  which  the  actual  gunnery  abilities  need 
to  be  applied  on  the  job  (listing,  description,  magnitude  and 
direction  of  each  such  deviation) 

B.  Probability  that  passing  the  test  under  the  specified  circum¬ 
stances  implies  the  learner  can  apply  the  gunnery  ability  under 
realistic  job  circumstances 

C.  Probability  that  failing  the  test  under  the  specified  circum¬ 
stances  implies  the  learner  cannot  apply  the  gunnery  ability  under 
realistic  job  circumstances 

D.  Deviations  between  the  tools  and  equipment  specified  for  the  tests 
and  the  tools  and  equipment  to  be  used  by  BIFV  gunners  to  apply 
the  tested  abilities  on  the  job  (listing  and  description  of  e.ach 
such  deviation) 

E.  Probability  that  passing  the  test  using  the  specified  tools  and 
equipment  implies  the  learner  can  apply  the  gunnery  ability  using 
the  tools  and  equipment  to  be  provided  on  the  job 

F.  Probability  that  failing  the  test  using  the  specified  tools  and 
equipment  implies  the  learner  cannot  apply  the  gunnery  ability 
using  the  tools  and  equipment  to  be  provided  on  the  job 

G.  Deviations  between  the  constraints  specified  for  the  tests  and  the 
actual  constraints  governing  the  BIFV  gunners'  application  of  the 
tested  abilities  on  the  job  (listing  and  description  of  each 

such  deviation) 

H.  Probability  that  passing  the  test  under  the  specified  constraints 
implies  that  the  learner  can  apply  the  gunnery  ability  under  the 
constraints  that  will  exist  on  the  job 

I.  Probability  that  failing  the  test  under  the  specified  constraints 
implies  that  the  learner  cannot  apply  the  gunnery  ability  under 
the  constraints  that  will  exist  on  the  job 


Figure  17.  (Continued) 


Standards  speci-  A.  Deviations  between  the  specified  test  standards  and  the  actual 

fitd  for  the  standards  of  adequate  performance  of  the  tested  abilities  on  the 

tailoring  tests  job  (listing,  description,  magnitude  and  direction  of  each  such 

deviation) 

B.  Probability  that  satisfaction  of  the  test  standards  implies  that  the 
learner  can  satisfy  the  actual  standards  of  adequate  performance 
on  the  job 

C.  Probability  that  failure  to  satisfy  the  test  standards  implies  that 
the  learner  cannot  satisfy  the  actual  standards  of  adequate  per¬ 
formance  on  the  job 


3.5.3  Production  of  appropriate  tests  for 

assessing  BIFV  gunner-learners' 
achievements  as  a  result  of  training 

Gunner-learner  A.  Relevant  achievements  for  which  tests  are  prepared  (listing  and 

achievements  for  description  of  each  such  achievement) 

which  tests  are  B.  Relevant  achievements  for  which  no  tests  are  prepared  (listing  and 

prepared  description  of  each  such  achievement) 

C.  Irrelevant  "achievements"  for  which  tests  are  prepared  (listing  and 
description  of  each) 

The  tests  prepared  D.  Elements  of  each  test  that  deviate  from  the  achievements  actually 
for  assessment  of  required  of  certified  BIFV  gunners  (listing  and  description  of 

the  BIFV  gunner-  each  such  deviating  element) 

learners'  achieve-  E.  Probability  that  each  test  will  be  passed  by  learners  who  actually 

ments  do  not  possess  the  achievement  being  tested 

F.  Probability  that  each  test  will  be  failed  by  learners  who  actually 
do  possess  the  achievement  being  tested 

G.  Cumulative  probability  that  an  actually  unqualified  learner  will 
pass  sufficient  achievement  tests  to  be  certified  as  a  gunner 

H.  Cumulative  probability  that  an  actually  qualified  learner  will 
fail  sufficient  achievement  tests  to  be  rejected  as  a  gunner 


3. 5. 3.1  Production  ot  achievement  tests  that 

provide  adequate  post-training  measures 
of  BIFV  gunner-learners'  abilities 


Domains  of  learn-  A.  Achievement  tests  that  can  only  be  passed  via  application  of  a 

ing  addressed  by  domain  of  learning  other  than  the  domain  relevant  to  the  ability 

the  tests  supposedly  to  be  tested  (listing  and  description  of  each  such  test) 

B.  Achievement  tests  that  can  also  be  passed  via  application  of  a 
domain  of  learning  bther  than  the  domain  relevant  to  the  ability 
supposedly  to  be  tested  (listing  and  description  of  each  such  test) 


Actions  elicited  C.  Deviations  between  the  actions  elicited  by  the  tests  and  the  actions 

by  the  tests  required  to  demonstrate  each  achievement  (listing  and  description 

of  each  such  deviation) 

D.  Probability  that  the  execution  of  the  elicited  action  implies  the 
ability  to  execute  the  action  actually  required  to  demonstrate 
the  achievement 

E.  Probability  that  a  failure  to  execute  the  elicited  action  implies 
the  inability  to  execute  the  action  actually  required  to  demon¬ 
strate  the  achievement 


Figure  17.  (Continued) 
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Objects  intended 
to  result  from  the 
actions  elicited 
by  the  tests 


3.5. 


Circumstances 
specified  for  the 
tests 


Tools  and  equip¬ 
ment  specified 
for  the  tests 


Constraints  spe¬ 
cified  for  the 
tests 


3.5. 


Standards  speci¬ 
fied  for  the 
achievement  tests 


F.  Deviations  between  the  objects  intended  to  result  from  the  test 
actions  and  the  objects  intended  to  result  from  the  actually 
required  actions  (listing  and  description  of  each  such  deviation) 

G.  Probability  that  the  production  of  the  test  object  implies  the 
ability  to  produce  the  actually  required  object 

H.  Probability  that  a  failure  to  produce  the  test  object  implies 
the  inability  to  produce  the  actually  required  object 

3.2  Production  of  appropriate  procedures 
for  applying  the  achievement  tests 
_ to  BIFV  gunner-learners _ 

A.  Deviations  between  the  circumstances  specified  for  the  tests  and 
the  circumstances  under  which  the  actual  tested  abilities  need  to 
be  applied  on  the  job  (listing,  description,  magnitude  and 
direction  of  each  such  deviation) 

B.  Probability  that  passing  the  test  under  the  specified  circum¬ 
stances  implies  the  learner  can  apply  the  gunnery  ability  under 
realistic  job  circumstances 

C.  Probability  that  failing  the  test  under  the  specified  circum¬ 
stances  implies  the  learner  cannot  apply  the  gunnery  ability  under 
realistic  job  circumstances 

D.  Deviations  between  the  tools  and  equipment  specified  for  the  tests 
and  the  tools  and  equipment  to  be  used  by  BIFV  gunners  to  apply 
the  tested  abilities  on  the  job  (listing  and  description  of  each 
such  deviation) 

E.  Probability  that  passing  the  test  using  the  specified  tools  and 
equipment  implies  the  learner  can  apply  the  gunnery  ability 
using  the  tools  and  equipment  actually  available  on  the  Job 

F.  Probability  that  failing  the  test  using  the  specified  tools  and 
equipment  implies  the  learner  cannot  apply  the  gunnery  ability 
using  the  tools  and  equipment  actually  available  on  the  job 

G.  Deviations  between  the  constraints  specified  for  the  tests  and  the 
actual  constraints  governing  the  BIFV  gunners'  application  of 
the  tested  abilities  on  the  job  (listing  and  description  of  each 
such  deviation) 

H.  Probability  that  passing  the  test  under  the  specified  constraints 
implies  that  the  learner  can  apply  the  gunnery  ability  under  the 
constraints  that  will  exist  on  the  job 

I.  Probability  that  failing  the  test  under  the  specified  constraints 
implies  that  the  learner  cannot  apply  the  gunnery  ability  under 
the  constraints  that  will  exist  on  the  job 


3.3  Production  of  appropriate  achievement 
test  standards  for  certifying 
_ learners  as  BIFV  gunners 

A.  Deviations  between  the  specified  test  standards  and  the  actual 
standards  of  adequate  performance  of  the  tested  abilities  on  the 
job  (listing,  description,  magnitude  and  direction  of  each  such 

deviation 

B.  Probability  that  satisfaction  of  the  test  standards  implies  that 
the  learner  can  satisfy  the  actual  standards  of  adequate  perfor¬ 
mance  on  the  job 

C.  Probability  that  failure  to  satisfy  the  test  standards  implies 
that  the  learner  cannot  satisfy  the  actual  standards  of  adequate 
performance  on  the  job 


Figure  17.  (Concluded) 
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5.  Concluding  Remarks  Concerning  the  Trial  Application  to 

Measurement  of  Training  System  t'erformance/Effectiveness 


Clearly,  much  remains  unproven  as  we  conclude  this  trial 
application.  Its  final  product  is  merely  a  partial  set  of  measures  for  assessing 
one  very  narrow  aspect  of  the  work  required  of  the  BIFV  Gunnery  Training 
Design  Subsystem.  Most  glaringly,  those  measures  are  untried.  We  have  not 
attempted  actually  to  apply  them,  nor  even  to  verify  that  the  information 
needed  to  generate  them  can  be  obtained.  Those  measures,  moreover,  derive 
from  attributes  that  also  are  unproven.  No  attempt  has  even  been  made  to 
demonstrate  convincingly  that  the  attributes  stated  really  constitute  all  of  the 
factors  pertinent  to  each  evaluative  issue,  and  only  those  factors. 

What,  then,  has  been  demonstrated?  If  nothing  else,  a  set  of 
measures  has  been  produced  that  definitely  is  issues-oriented.  We  have  been 
able  to  proceed  from  a  framework  of  basic  performance  requirements  to  a 
statement  of  exactly  what  requirements  apply  to  a  given  measurement 
application.  We  have  shown  that  such  requirements  can  be  expressed  in  terms 
that  permit  concrete,  tractable  attributes  to  be  defined.  None  of  the 
attributes  stems  from  any  particular  approach  to  training  system  design.  All 
of  the  attributes  strictly  derive  from  the  performance  requirements  that  are 
common  to  all  such  designs.  This  is  a  critical  point:  it  demonstrates  that  the 
APM  at  least  is  rooted  firmly  in  valid  measurement  issues.  Too  often, 
evaluations  of  new  systems  are  clouded  by  irrelevant  and  unfair  comparisons 
with  older  design  concepts.  The  basic  question  of  whether  the  new  system 
does  what  it  is  supposed  to  do  sometimes  is  overlooked  because  the  evaluators 
actually  measure  how  the  new  system  "looks"  in  comparison  to  older  models. 
Conclusions  based  on  such  evaluations  typically  boil  down  to  this:  "The 
system  is  no  good  because  we've  never  done  it  this  way  before."  That 
design-oriented  philosophy  of  measurement  suppresses  technological 
breakthroughs.  In  the  training  sphere,  it  prevents  modern  and  effective 
methods  of  work  site,  learner-tailored  instruction  from  playing  their 
appropriate  role  in  the  total  learning  environment  and  it  masks  the  relative 
ineffectiveness  of  the  traditional,  institutonal  methods  of  training.  The 
authors  consider  it  to  be  a  major  step  forward  to  have  shown  that  a  training 
measurement  scheme  can  be  kept  free  of  artificial  judgments  of  design.  Use  of 
the  APM  has  forced  a  rather  complete  taxonomy  of  performance  requirements  to 
be  considered  in  a  measurement  application;  more  complete  than  a  typical, 
less-structured  analysis  would  provide. 

Finally,  it  has  been  shown  that  measures  can  be  identified  for  these 
performance-oriented  attributes.  Much  remains  to  Be  done  to  develop 
guidelines  for  insuring  that  the  best  such  measures  always  are  identified.  But 
the  first  step,  at  least,  has  been  taken  in  that  direction. 

The  next  section  of  this  report  explores  the  utility  of  the  APM  for 
specifying  how  training  system  performance  requirements  can  be  met. 
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The  APM  for  Development  of  Training  System  Requirements 


The  purpose  of  this  section  is  to  illustrate  how  training  system 
performance  taxa  identified  for  any  particular  human-machine  system  can  be 
applied  to  produce  training  system  requirements  for  that  particular  system. 
The  approach  presented  proceeds  through  the  following  steps: 

(1)  Define  precisely  the  system  of  interest. 

(For  this  example,  the  system  of  interest  will  be  one  training 
subsystem,  i.e.,  the  training  design  subsystem  of  an  Army 
training  system  under  consideration  for  further  development/ 
procurement. ) 

(2)  Determine  and  identify  training  system  performance  requirements. 

(This  example  will  illustrate  the  procedures  to  be  used  in 
translating  general  training  design  performance  taxa  for  any 
system  into  training  system  requirements  for  a  system  of 
interest. ) 

(3)  Present  an  approach  on  how  these  training  subsystem  design 
requirements  may  be  satisfied  by  the  training  design  subsystem. 

(Here  examples  are  presented  to  illustrate  how  the  training 
design  subsystem  plans  its  approach  to  satisfy  the  training 
design  requirements.) 

1 .  The  System  of  Interest:  The  Design  Subsystem  of 
0.5,  Army  Handgun  Training 

Currently,  several  U.S.  joint  military  services  study  groups  have 
been  formed  to  determine  the  need  and  feasibility  of  adopting  a  standardized 
9rnm  handgun  as  a  replacement  for  more  than  25  different  makes,  models  and 
types  of  handguns  currently  in  the  U.S.  services  inventory  (Air  Force,  Army, 
Navy,  Marines  and  Coast  Guard).  A  Joint  Service  Small  Arms  Training  Task 
Force  (JSSATTF)  has  recommended  that  all  U.S.  armed  services  adopt  a  single 
family  of  9  mm  handguns;  however,  to  date  no  final  conclusions  have  been 
made.  The  U.S.  Army  currently  requires  the  largest  number  of  handguns 
(approximately  331,000)  as  a  personal  defensive  weapon  for  such  Army 
personnel  as  certain  front-line  officers,  non-commissioned  officers,  operators  of 
crew-served  weapons,  aircrews,  drivers,  equipment  operators,  rear  echelon 
personnel,  etc. 

In  the  context  of  this  application  an  assumption  is  made  that  the 
JSSATTF  would  assign  the  overall  responsibility  for  training  development  to 
the  Army.  Since  the  new  handgun  is  considered  to  be  in  the  family  of 
Infantry  weapons,  the  U.S.  Army  Infantry  School  would  develop  the  curricula 
for  the  Army  9mm  Handgun  Training  System  (AHTS).  The  AHTS  is  a  training 
system  that  incorporates  the  six  general  training  subsystems  within  itself; 
these  are  the  Command,  Design,  Enabling,  Emplacement,  Logistics  and  Delivery 
subsystems.  Figure  18  depicts  the  network  of  other  systems  that  interact  with 
the  AHTS.  These  include: 


The  system  known  as  the  U.S.  Army  Infantry  School.  At  this 
stage  of  training  development,  the  AHTS  would  be  wholly 
contained  within  the  Infantry  School,  which  would  be  considered 
a  suprasystem  to  the  system  of  interest. 

JSSATTF.  The  Task  Force  is  comprised  of  highly  qualified 
representatives  of  each  of  the  U.S.  military  services.  The 
JSSATTF  will  determine  likely  handgun  users  and  uses.  The 
JSSATTF  will  develop  the  training  system  requirements  for  the 
AHTS.  The  JSSATTF  is  a  suprasystem  to  the  system  of 
interest. 

U.S.  military  services.  These  macrosystems  impact  on  training 
system  requirements  through  the  JSSATTF.  The  AHTS  has 
to  satisfy  their  training  system  requirements.  Eventually,  each 
of  these  services  will  require  its  own  service-specific  handgun 
training  system.  They  will,  in  fact,  have  a  network  of  training 
systems  which  will  be  somewhat  analogous  to  the  AHTS,  i.e., 
various  MOS  training  systems,  unit  training  systems,  school 
training  systems,  etc. 

Various  U.S.  Army  Branch  School  Training  Systems  (Armor, 
Artillery,  Signal,  etc.).  The  AHTS  will  be  integrated 
eventually  into  these  training  systems.  AHTS  training  must  be 
compatible  with  these  training  systems.  These  systems 
therefore  must  be  considered  as  collateral  systems  to  the  system 
of  interest  and  the  AHTS. 

Various  U.S.  Army  Training  Systems.  The  personnel  who  will 
be  assigned  the  new  handgun  function  within  a  specific  military 
occupational  specialty  (MOS).  The  AHTS  must  be  cdmpatible 
with  each  of  these  training  systems.  These  personnel  will  be 
the  AHTS  learners.  The  systems  are  collateral  to  the  AHTS. 

Various  types  of  U.S.  Army  unit  training  systems  (mechanized 
infantry,  self-propelled  artillery,  heavy-construction  engineer, 
etc.).  The  AHTS  learners/users  are  within  these  types  of 
training  systems  as  depicted  by  their  Tables  of  Organizational 
Equipment  (TOE).  The  AHTS  must  be  compatible  with  the 
other  on-going  training  the  AHTS  learners  have  or  will  have 
received.  These  systems  are  collateral  to  the  AHTS. 

The  9mm  handgun  itself  with  its  characteristics  is  an  essential 
collateral  system  to  the  AHTS. 


The  Army  Handgun  Training  Design  Subsystem  (AHTDS),  a 
subsystem  of  the  AHTS,  is  the  system  of  interest.  The  AHTDS  is  given  the 
training  requirements  identified  by  the  JSSATTF  and  the  AHTS  Command 
subsystem  and  translates  these  requirements  into  specific  training 
plans/sp.  rifications.  The  AHTDS  is  the  curriculum  developer.  As  such,  the 
AHTDS  determines  the  learner  behaviors  that  are  to  be  affected  during 
learning,  and  specifies  the  sensory  activities  to  be  conducted  to  achieve  the 
effects.  The  AHTDS  produces  all  of  the  learning  procedures  and  resource 
specifications  and  some  of  the  material  required  by  the  AHT  Delivery 
subsystem.  The  AHTDS  makes  basic  decisions  with  regard  to  learning 
facilitators,  facilities,  material  and  equipment  needed  to  implement  the  learning 
activities . 

2 .  Determining  Training  System  Performance  Requirements 

At  the  conceptual  stages  of  a  particular  training  system's 
development  the  suprasystem  (which  has  direct  administrative  control  of  the 
training  system),  has  the  overall  responsibility  to  develop  training  system 
performance  requirements  for  each  of  the  training  system's  six  subsystems 
(Command,  Design,  Enabling,  Emplacement,  Logistics  and  Delivery).  For  this 
application,  the  JSSATTF  is  the  suprasystem  and  therefore  the  training 
administrator  and,  as  such,  exercises  administrative  control  over  the  total 
training  system,  i.e.,  the  AHTS.  The  JSSATTF  must  specify  the  performance 
requirements  in  such  detail  as  to  insure  that  the  total  training  system  achieves 
its  purposes  of  learning  and  helping-to-learn . 

This  application  focuses  on  the  Training  Design  Subsystem  of  the 
AHTS.  In  developing  training  performance  requirements,  the  JSSATTF  can 
apply  the  general  design  subsystem  performance  taxa  as  shown  in  Figure  7. 
Each  taxon  is  a  performance-oriented  Design  subsystem  requirement, 
'therefore,  in  developing  the  Design  subsystem  training  performance 
requirement;,  the  complete  taxa  shown  in  Figure  7  applies.  The  taxa 
constitute  a  performance  checklist  of  capabilities,  procedures  and  products  that 
the  Design  subsystem  must  plan  for  in  meeting  the  training  systems  purposes. 

The  JSSATTF  must  translate  the  general  Design  subsystem  training 
performance  requirements  specifically  for  the  AHTS.  At  this  stage  of  system 
development,  the  JSSATTF  knows  who  the  learners  are,  (i.e.,  the  Army 
handgun  users)  and,  generally,  for  what  purposes  the  Army  handgun  is  to  be 
used,  (i.e.,  the  jobs  for  which  the  handgun  is  to  be  employed). 

As  an  aid  in  writing  the  AHT  Design  Subsystem  (AHTDS)  training 
requirements,  it  is  helpful  to  organize  the  performance  taxa  identified  in 
f  igure  7  into  logically  associated  groups  within  a  particular  contextual  aspect 
of  system  performance,  e.g.,  Performance  Potentialities.  A  specific  objectives- 
level  taxon  should  be  followed  by  the  first  functional  purposes-level  taxon;  the 
first  functional  purposes-level  taxon  should  then  be  followed  by  its  supporting 
characteristics-level  taxa.  The  process  is  continued  for  that  objective  level 
taxon  until  all  of  the  functional  purposes-level  taxa  and  their  supporting 
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characteristics-level  taxa  have  been  tied  together  explicitly.  For  example,  to 
prepare  the  AHTDS  training  performance  requirements,  the  taxa  for 
Performance  Potentiality  1.4  shown  in  Figure  7  could  be  organized  as  follows: 


Objectives  Level 

1.4 

Defining  training  content 

Functional  Purposes  Level 

1.4.1 

Selecting  stimuli  for  each 
objective 

Characteristics  Level 

1.4. 1.1 

Defining  propositions,  names, 
facts,  etc.,  relevant  to 
information  objectives 

1.4. 1.2 

Defining  concepts,  rules, 
algorithms,  etc.,  relevant  to 
mental  skills  objectives 

1.4. 1.3 

Defining  movements,  timings, 
actions,  etc.,  relevant  to  physical 
skills  objectives 

1.4. 1.4 

Defining  values,  choices,  etc., 
relevant  to  attitude  objectives 

Functional  Purposes  Level 

1.4.2 

Selecting  instructional  setting  for 
each  objective 

Characteristics  Level 

1.4.2. 1 

Assessing  applicability  of  Job 
Performance  Aids  (JPA's) 

1.4. 2. 2 

Assessing  applicability  of 

Self-Teaching  Exportable  Packages 
(STEP'S) 

1.4. 2. 3 

Assessing  applicability  of  Formal 
On-the-Job  Training  (FOJT) 

1.4. 2. 4 

Assessing  applicability  of 

Installation  Support  School  (ISS) 

1.4. 2. 5 

Assessing  applicability  of  Resident 
School 

Etc. 

Use  of  this  or  some  other  suitable  scheme  greatly  facilitates 
organization  of  the  performance  taxa  needed  to  generate  a  particular  training 
subsystem's  performance  requirements. 


The  next  step  is  to  translate  the  organized  general  performance 
taxa  into  subsystem  training  performance  requirements  for  a  specific  training 
system.  Again  for  this  application,  training  performance  requirements  are  for 
the  AHTDS.  Each  requirement  must  address  the  specific  subsystem  which 
must  perforin  the  work.  Each  of  the  contextual  stage  performance  taxa 
systematically  will  identify  the  performance  requirements  which  must  be 
addressed  by  the  training  subsystem.  Generally,  the  contextual  stage 
requirements  for  the  AHTDS  will  state  the  following: 

(1)  AHTDS  Performance  Potentialities  Requirements 

These  requirements  state  the  capabilities/potentialities  the 
AHTDS  must  address. 

(2)  AHTDS  Performance  Processes  Requirements 

These  requirements  state  the  processes/procedures  the  AHTDS 
must  address. 

(3)  AHTDS  Performance  Froaucts  Requirements 

The  requirements  state  the  products  the  AHTDS  must  address. 

To  continue  the  above  example,  the  performance  taxa  for 
Performance  Potentiality  1,4  and  Functional  Purpose  1.4.1  could  be  restated 
into  AHTDS  training  performance  requirements  as  follows: 

Objectives  Level  1.4  The  Handgun  Training 

Design  Subsystem  must 
possess  the  ability  to  define 
the  handgun  training  content 
required  to  achieve  the 
user's  performance- objectives 

Functional  Purposes  Level  1.4.1  Given  a  list  of  handgun 

users  and  uses,  and  given 
the  defined  training  gotls 
and  analyzed  performance 
objectives,  the  Design 
Subsystem  must  be  able  to 
select  stimuli  for  each 
performance  objective 
relevant  to  the  domains  of 
learning 

Characteristics  Level  1.4.  1.1  The  Handgun  Training 

Design  Subsystem  must 
demonstrate  that  it  possesses 
the  ability  to  define 
propositions,  facts,  names, 
etc.,  that  are  relevant  to 
information-type  objectives 


-99- 


1.4.1..’ 


The  Handgun  Training 
Design  Subsystem  must 
demonstrate  that  it  possesses 
the  ability  to  define 
concepts,  rules,  algorithms, 
etc.,  relevant  to  mental 
skills  objectives 

1.4. 1.3  The  Handgun  Training 

Design  Subsystem  must 
demonstrate  that  it  possesses 
the  ability  to  define  handgun 
user/uses  movements, 
actions,  timings,  etc.,  for 
relevant  physical  skill 
objectives 

1.4. 1.4  The  Handgun  Training 

Design  Subsystem  must 
demonstrate  that  it  possesses 
the  ability  to  define  handgun 
user /uses  choices,  values, 
etc.  ,  relevant  to  attitude 
objectives 

It  should  be  noted  by  the  reader  that  each  requirement  in  the  above 
example  is  specific  to  the  A11TDS.  Further,  since  these  are  Performance 
Potentialities  Requirements,  they  are  stated  in  terms  of  "abilities"  or 
''capabilitier."  which  the  AHDTS  must  demonstrate  that  it  possesses  to 
effectively  respond  to  the  requirement.  Similarly,  the  Performance.  Process 
Requirements  are  stated  in  terms  which  require  the  AHTDS  to  demonstrate  Its 
ability  to  effectively  plan  an  approach  to  satisfy  these  requirements.  With 
regard  to  Performance  Product  Requirements,  these  are  stated  in  terms  which 
require  the  AHTDS  to  demonstrate  its  ability  to  plan  how  it  will  deliver  or 
produce  the  specified  product  requirements.  Figure  19  is  a  sample  of  detailed 
training  performance  requirements  specific  to  the  AHTDS.  Figures  B1  through 
B6  present  all  of  the  training  performance  requirements  for  the  basic 
potentialities  specific  to  the  AHTDS. 

3.  An  Approach  to  Developing  Training  System 

Design  Specifications 

The  general  training  system  performance  taxa  shown  in  Figures  7 
and  8  constitute  the  performance  requirements  faced  by  every  Design 
Subsystem  and  Enabling  Subsystem,  respectively .  Figures  19  and  B1-B6  in 
Appendix  B  present  Design  Subsystem  taxa  in  terminology  that  is  specific  to 
the  Army  Handgun  Training  System.  Those  figures,  too,  deal  in  performance 
requirements .  In  earlier  segments  of  this  report,  the  point  was  emphasized 
tKat  system  evaluation  should  address  itself  solely  to  issues  stemming  from 


Performance  Potentiality  Hierarchy  Number  I:  The  capability  of  Identifying  training  goala  and  priorities 


1.1 

TTTc  Handgun  Training  Design  Subsystem 
must  possess  the  ability  *9  identify  train¬ 
ing  goals  and  priorities  for  all  intended 
military  users  and  uses  of  the  handgun. 


1.1.1 

Civen  s  ha*  of  handgun  users 
and  uses,  the  Design  Subsystem 
must  be  able  to  define  the 
•cope  of  learning  needed  for 
each  handgun  uaer/use  in  terms 
of  the  knowledge,  skills,  and 
attitudes  required. 


irhe  Handgun  Training 
Design  5  ubsystem  must 
demonstrate  that  It  pos¬ 
sesses  the  ability  to 
identify,  for  each  hand¬ 
gun  uaer/use,  ssactly 
•That  the  user  must  know; 
exactly  what  skills  the 
user  must  possess:  and 
exactly  what  attitudes 
the  user  must  manifest. 

UU 

The  Handgun  Training 
Design  Subsystem  must 
demonstrate  that  it  pos¬ 
sesses  the  ability  to 
analyze  the  identified 
knowledge,  skills,  and 
attitudes  to  determine 
their  suitability  for 
training . 


1.1,2 

Given  a  list  of  handgun  users 
and  uses,  the  Design  Subsystem 
must  be  able  to  define  what 
each  user  ultimately  will  be 
able  to  do  with  the  handgun, 
tn  training  goal  terms. 


1.1. 2.1 

The  hand g  un  Training 
Design  Subsystem  must 
d.monstrate  that  it  pos- 
Haua  the  ability  to 
define  the  neceaaary 
fev.la  of  performance 
achievement,  that  each 
handgun  uacr  must  mani- 
faat  in  order  to  apply 
the  handgun  to  ita 
Intended  use,, 

1 .1.2.2 

the  Handgun  Training 
Deaign  Subsystem  must 
demonstrate  that  it  ia 
able  to  determine  cor¬ 
rectly  the  pre-training 
levele  of  the  relevant 
performance  achicvementa 
poeeeaaed  by  membere  of 
the  intended  handgun  user 
populattona. 


1.1.? 

Given  a  Uet  of  handgun  uaera 
and  uaee,  and  gtvan  the  train¬ 
ing  goala  defined  (or  each 
uaer/uae,  the  De-sign  Subeyetem 
muat  be  able  to  identify  the 
relative  Importance  of  esch 
defined  goal. 


1, 1.1.1 

The  Handgun  Training 
Design  Subsystem  muat 
demonstrate  that  it  has 
the  ability  to  formulate 
assessments  of  training 
goal  importance. 


1.1 . 3.2 

The  Handgun  Training 
Deaign  Subsystem  must 
demonstrate  that  it  can 
devise  or  select  a  suit¬ 
able  scheme  for  ranking 
the  relative  importance 
of  ita  training  goala. 


1.1.4 

Given  a  list  of  handgun  users 
and  uses,  and  given  the  train 
ing  goala  defined  for  each 
uaer/uae,  the  Design  Subsystem 
must  be  able  to  express  the 
goals  In  behaviorally  oriented 
terms  to  establish  a  basis  for 
Identifying  handgun  user  per¬ 
formance  objectives. 


The  Handgun  Training 
Dosign  Subsystem  mutt 
demonstrata  that  it  can 
devise  or  select  e  suit¬ 
able  taxonomy  of  goals/ 
objectives  to  be  used  to 
classify  Ita  intended 
training  outcomes. 


lh«  Handgun  Training 
Deaign  Subsystem  moat 
demonstrate  that  it  ia 
capable  of  slating  Ita 
goala  tn  terms  that  per¬ 
mit  the  neceaaary  user 
behaviors  to  be  Identified. 


1.1.2.) 


The  handgun  Training 
Design  Subsystem  muat 
demonstrate  that  tt  pos¬ 
sesses  the  ability  to 
discern  correctly  the 
discrepancies  axiating 
between  current  and  re¬ 
quired  levels  of  relevant 
performance  achievements 
within  the  intended  hand¬ 
gun  user  populations. 


Figure  19.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Potentialities  Needed:  Hierarchy  Number  1 
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performance  requirements.  The  evaluator’s  proper  frame  of  mind  can  be 
illustrated  as  follows:  "I  don't  care  what  it  looks  like,  and  I  don’t  care  why  it 
looks  that  way;  1  only  want  to  know  if  it  performs  the  way  it  is  supposed  to 
perform."  The  evaluator  needs  to  have  that  point  of  view  in  order  to  insure 
that  the  evaluation  process  and  its  conclusions  are  not  contaminated  by 
irrelevant  considerations  of  design .  The  evaluator  cannot  allow  himself  or 
herself  to  jump  to  the  con  elusion  "  that  a  system  is  no  good  simply  because  it 
"looks"  different  from  its  predecessors.  Neither  can  the  evaluator 
automatically  assume  that  the  system  is  good  simply  because  it  embodies 
radically  new  design  concepts.  The  evaluator  needs  to  preserve  a  strict 
disinterest  in  issues  of  design. 

homebody,  however,  does  have  to  worry  about  what  the  system 
should  "look  like."  Somebody  has  to  build  the  system  to  satisfy  its 
performance  requirements.  If  it  isn't  built  properly,  it  surely  won't  meet 
those  requirement::..  And  the  proper  building  of  anything  has  to  start  with 

the  proper  design . 

The  specification  of  a  training  system  design  is  nothing  more  nor 
less  than  the  plan  for  building  the  training  system  to  satisfy  its  performance 
requirements.  The  design  specification  thus  must  evolve  from  those 
requires  ants ,  i.c.,  from  the  same  taxa  of  potentialities,  processes,  and 
product:.,  from  which  the  evaluator  extracts  the  measures  of  system  performance 
and  effectiveness.  The  system  designer  looks  at  each  taxon  and  asks,  "How 
can  I  see  to  it  that  the  system  will  have  all  of  these  potentialities?  How 
should  I  arrange  for  the  system  to  carry  out  all  of  these  required  processes? 
How  can  I  best  insure  that  it  will  deliver  all  of  these  required  products?"  Of 
course,  the  designer  must  come  up  with  answers  to  these  questions  that  are 
compatible  with  the  taxa  of  Environment  and  Constraints  under  which  the 
system  must  operate.  But  the  questions  themselves  are  rooted  in  the  taxa  of 
performance. 


Our  question  is:  How  does  the  designer  answer  these  questions? 
How  does  a  design  specification  evolve  from  the  system  performance 

requirements? 

The  first  step  in  this  process  is  identical  to  the  first  step  in  the  , 
evaluation  process:  one  examines  each  taxon  to  identify  its  associated  system 
attributes.  Here  again,  an  attribute  is  some  concrete  element  of  the  system, 
something  that  car  be  seen,  examined,  submitted  to  analysis,  etc.  It  is, 
therefore,  something  for  which  concrete  plans  may  be  formulated. 

■The  Design  attributes  associated  with  a  particular  taxon  are  any  and 
all  features  of  the  system's  people,  equipment,  and  procedures  that  may  affect 
satisfaction  T  the  performance  requirement  represented  by  that  taxon.  The 
features  of  relevance,  of  course,  may  vary  from  one  taxon  to  another.  The 
relevant  features  of  people  associated  with  one  taxon,  for  example,  might 
include  their  visual  acuity,  their  physical  strength,  their  height,  their 
resistance  to  motion,  sickness,  etc.  That  is,  the  nature  of  the  particular 
performance  -equiroment  would  be  such  that  the  system  would  need  to  have 
some  operators  possessing  those  physical  attributes  if  the  system  is  to  be  able 
to  meet  that  requirement.  Analysis  of  some  other  taxon  might  disclose  that 
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people  are  needed  who  possess  a  certain  degree  of  knowledge  about  some 
particular  subject,  who  can  apply  certain  mental  skills,  and  who  manifest 
certain  attitudes  concerning  particular  topics.  The  nature  of  that  particular 
performance  requirement  would  demand  that  it  be  addressed  by  people  who  had 
those  specific  intellectual  and  affective  attributes. 

The  system  designer's  job  thus  begins  with  the  identification  of  all 
of  the  features  of  people,  equipment,  and  procedures  that  relate  to  the 
satisfaction  of  all  of  the  performance  requirements  that  the  system-to-be-built 
will  face.  Then,  the  designer  must  prepare  plans  or  statements  showing 
exactly  how  he  or  she  will  see  to  it  that  all  of  those  features  are  included  in 
the  system. 

To  illustrate,  recall  that  one  performance  requirement  that  a  training 
system  designer  faces  is  job  analysis.  That  is  an  objectives-level  taxon  of 
process.  The  designer  knows  that  the  plans  for  building  any  training  system 
must  include  a  provision  to  "analyze  the  job  for  which  training  is  to  be 
developed."  The  problem  is  that  the  form  or  method  of  job  analysis  that  may 
work  well  for  one  training  system  may  not  be  most  appropriate  or  even 
workable  f<  r  another  training  system.  A  designer  who  doesn't  pause  to 
consider  the  attributes  of  the  specific  system  to  be  built  might  simply  select  a 
job  analysis  plan  "off  the  shelf,"  dust  it  off  and  incorporate  it  into  the  system 
specification.  That  plan  might  place  heavy  emphasis  on  the  conduct  of 
observations  of  actual  performances  of  the  job  to  be  analyzed.  That  is 
certainly  a  traditional  and  often  effective  approach  to  job  analysis.  But  in  the 
case  of  a  brand  new  job,  never  before  performed,  it  may  be  totally 
inappropriate  to  rely  on  observational  methods  of  job  analysis:  there  simply 
wouldn't  be  any  job  performers  to  observe.  A  study  of  the  attributes  of  the 
data  available  for  the  job  analysis  would  disclose  that  other  methods  -would  be 
required,  e.g.,  methods  relying  on  review  of  the  specifications  for  the  newly 
developed  job,  interview  with  the  designers  of  the  new  job,  etc.  Similarly, 
the  training  system  designer  would  determine  that  the  people  who  will  be 
assigned  to  perform  the  job  analysis  would  need  to  be  skilled  in  interviewing 
techniques  and  "paper-and-pencil"  analytic  methods,  rather  than  simply  being 
well  trained  observers. 

Figure  20  presents  a  sample  of  the  design  specification  that  might  be 
written  in  response  to  the  performance  requirements  of  the  Army  Handgun 
Training  System.  The  selected  sample  addresses  the  first  hierarchy  of 
performance  potentialities  of  the  Design  Subsystem,  i.e.,  the  objectives, 
functional  purposes,  and  characteristics  taxa  associated  with  the  potential  for 
identifying  training  goals  and  priorities.  Appendix  B  provides  additional 
samples  of  design  specifications  for  the  process  and  products  aspects  of  the 
AHTDS. 

Preparation  of  a  design  specification  such  as  that  shown  in  Figure  20 

"merely"  is  a  matter  of  deciding  how  best  to  insure  that  all  appropriate 

features  of  people,  equipment,  and  procedures  are  built  into  the  developing 
system.  Of  course,  a  good  deal  of  design  talent  is  needed  to  do  "merely" 

that.  Good  system  designers  are  paid  well,  and  deserve  to  be.  Deciding  how 

a  set  of  diverse  performance  requirements  can  be  met  most  effectively  is  never 
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Potentiality  Performance  Requirement  Number  Is 
The  capability  of  identifying  training  goals  and  priorities. 


(1.1) 

The  Army  Handgun  Training  Design  Subsystem  Task  Force 
recognized  that  a  solid  capability  for  identifying  its  training  goals  and 
establishing  their  relative  importance  must  be  brought  to  bear  on  the 
training  design  effort.  The  Task  Force  has  assembled  a  highly  qualified 
team,  including  representatives  of  all  user  services  whose  members 
collectively  possess  skills  and  experience  in  such  relevant  disciplines  as 
task  analysis,  operations  research,  human  factors  engineering,  and 
training  program  design.  The  members  of  the  team  will  have  access  to  all 
relevant  documentation  on  the  handgun,  and  will  have  ample  opportunity 
to  interview  current  and  prospective  users  of  the  handgun,  including 
some  Allied  military  personnel.  The  team  will  develop  separate  lists  of 
training  goals  and  priorities  for  each  service's  intended  user  populations, 
as  well  as  a  master,  all-services  list.  Names  and  relevant  experience  of 
the  team  members  are  given  in  Section  **  of  this  specification. 

(1.1.1,  1.1.2,  1.1.3,  1.1.4) 

The  Task  Force  has  determined  that  the  principal  use  of  the 
handgun  will  be  as  a  short-term  personal  defensive  weapon.  Users  who 
will  so  employ  the  handgun  include  front  line  officers  and 
non-commissioned  officers;  operators  of  crew-served  weapons;  pilots; 
drivers;  various  equipment  operators  and  technicians;  and,  certain  rear 
echelon  personnel.  A  secondary  use  of  the  handgun  will  be  as  a  law 
enforcement  offensive  and  defensive  weapon,  to  be  used  by  military 
police.  The  charter  of  the  task  Force  explicitly  excludes  the  latter  use 
from  the  scope  of  the  training  design  requirements.  Accordingly, 
members  of  the  team  will  focus  exclusively  on  the  handgun's  use  as  a 
personal  defensive  weapon.  They  will  analyze  a  representative  set  of 
close-contact-combat  reports  from  World  War  II  and  the  Korean  and  Viet 
Nam  conflicts  to  develop  scenarios  in  which  the  handgun  user  tasks  can 
be  identified.  Appropriate  methods  of  task  analysis  will  be  applied  to 
determine  handgun  user  usage  requirements  in  those  scenarios,  and  to 
identify  the  knowledge,  skills,  and  affective  characteristics  needed  to 
satisfy  those  usage  requirements.  Using  the  Delphi  and/or  other  suitable 
rating  techniques,  the  team  members  will  determine  the  criticality  of  the 
knowledge,  skill,  and  attitude  ingredients.  This  will  be  coupled  with  a 
review  of  the  intended  user  populations  to  assess  the  current  statics  of 
the  knowledge,  skills,  and  attitudes.  The  critical  deficiencies  thus 
identified  will  serve  as  the  basis  for  identifying  the  training  goals.  The 


Figure  20.  Sample  Design  Specifications  for  the  AHTDS 
(Potentialities  Aspect) 

[Appendix  B  provides  sample  design  specifications,  for  the  Processes  and 
Products  Aspects  of  the  AHTDS) 
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goals  will  be  evaluated  in  terms  of  their  relative  importance;  this  step  will 
solicit  input  from  representatives  of  the  intended  user  populations  as  well 
as  from  the  team  members  individual  and  collective  judgments.  The  team 
members  will  apply  their  human  factors  and  training  development  expertise 
to  insure  that  the  goals  express  the  quantifiable  actions  and  conditions 
required  for  effective  use  of  the  handgun. 

(1.1. 1.1,  1.1. 1.2) 

Analysis  of  the  knowledge,  skills,  and  attitudes  required  for 
effective  use  of  the  handgun  for  personal  defense  will  proceed  in 
accordance  with  the  information-decision-action  attribute  model,  described 
in  detail  in  Section  **  of  this  specification.  Briefly,  this  model  calls  for 
analysis  of  the  sequential  operations  required  of  a  task  under  study. 
The  operations  are  examined  at  steadily  increasing  levels  of  detail  until  all 
human  actions,  movements,  queries,  decisions,  etc.,  have  been  identified, 
together  with  their  stimuli  and  outputs.  Increasingly  detailed 
formulations  of  the  human  knowledge  and  skill  requirements  emerge  from 
this  analysis,  and  operator  attributes  and  attitudinal  characteristics  can 
be  inferred  from  those  requirements.  The  team  will  evaluate  each 
identified  requirement  to  determine,  first,  the  need  for  training  to  insure 
that  it  will  be  satisfied  by  intended  handgun  users;  second,  the  expected 
amount  and  cost  of  training  that  would  be  needed  to  insure  satisfaction  of 
the  requirement;  and,  third,  the  practicality  of  providing  that  training. 
Any  knowledge,  skill,  attribute,  or  attitudinal  requirement  not  deemed 
suitable  for  training  will  be  established  as  prerequisites  for  candidate 
handgun  learners. 

(1.1. 2.1,  1.1. 2. 2,  1.1. 2. 3) 

The  team  sill  examine  the  required  knowledge,  skills,  and  attitudes 
in  the  context  of  the  representative  scenarios.  The  purpose  will  be  to 
assess  the  achievement  standards  required  for  effective  use  of  the 
handgun  as  a  personal  defense  weapon.  Factors  such  as  accuracy, 
repeatability,  duration,  etc.,  of  knowledge /skill  performance  will  be 
varied  to  determine  their  impact  on  the  scenarios'  outcomes.  Attitudinal 
and  attribute  characteristics  also  will  be  varied  and  assessed.  Minimum 
standards  of  achievement  then  will  be  postulated.  Representatives  of  the 
intended  user  populations  will  be  examined  to  determine  their  current 
levels  of  achievement.  These  representatives  will  form  a  sample  of  users 
stratified  by  characteristics  relevant  to  handgun  usage,  e.g.,  such 
characteristics  as  the  user's  job  assignment,  anthropometric 
characteristics,  GS  scores,  and  other  relevant  factors  that  may  have 
emerged  from  the  analysis.  Findings  will  be  compared  with  the  identified 
minimum  achievement  standards,  to  discern  achievement  discrepancies  that 
are  to  be  addressed  in  training.  This  will  produce  o  refinement  of  the 
training  needs  analysis  discussed  above. 


Figure  20.  (Continued) 
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(1.1.3. 1.  1. 1.3.2) 


The  training  requirements  emerging  from  the  team's  analysis  assessed 
as  to  relative  importance.  Team  members  will  produce  individual, 
independent  ratings  for  the  training  needs  identified  for  their  own 
services'  intended  users.  These  individual  ratings  will  be  submitted 
anonymously,  and  will  be  reviewed,  discussed,  and  critiqued  in  a 
round-table  format.  That  will  be  done  first  on  an  individual  service  basis 
and  finally  collectively  for  all  users/uses.  Consensus  sets  of  needs 
importance  ratings  will  emerge  from  this  activity.  Finally,  the  needs  list 
will  be  reviewed  and  rated  by  the  representative  intended  users  and  by 
selected  Allied  current  handgun  users,  for  validation  and  refinement. 

(1.1. 4.1,  1.1. 4. 2) 

Each  selected  training  goal  will  be  classified  as  to  its  relevant  domain 
of  learning.  The  tripartite  taxonomy  of  B.S.  Bloom  (1956)  will  be  used 
for  this  purpose,  i.e.,  cognitive-affective-psychomotor .  Within  each 
domain,  the  goals  will  be  expressed  as  observable  actions  which  the 
handgun  learners  will  demonstrate  under  defined  conditions. 


Figure  20.  (Concluded) 


easy,  and  may  demand  considerable  levels  of  such  difficult-to-instill  qualities 
as  intuition,  creativity,  and  insight.  System  design  may  well  be  more  science 
than  art,  but  good  system  design  is  always  the  province  of  artful  scientists. 


The  authors  make  no  claim  to  artistry,  nor  to  possession  of  the 
creative  or  insightful  instincts  of  the  training  designer.  The  sample  design 
specification  given  in  Figure  20  undoubtedly  is  deficient  in  numerous  respects. 
In  one  major  respect,  however,  it  is  a  shining  example:  it  addresses  system 
performance  requirements  and  nothing  but  performance  requirements.  Experts 
in  handgun  training  possibly  could  look  at  Figure  20  and  uncover  instances 
where  the  offered  proof  of  the  ability  to  identify  training  goals  is  weak, 
inconclusive,  or  even  incompetent.  We  submit,  however,  that  those  design 
specifications  do  not  overlook  even  one  aspect  of  the  constituents  of  that 
ability,  i.e.,  the  specifications  at  least  offer  some  proof  for  everything  that 
needs  to  be  proven.  Neither  do  the  specifications  deal  with  anything  that  is 
not  germain  to  the  ability  to  identify  training  goals.  Our  design  specifications 
are  firmly  rooted  in  the  performance  requirements  that  the  Army  Handgun 
Training  System  really  faces.  The  APM  provided  the  capability  of  identifying 
those  requirements.  As  such,  it  is  a  valuable  tool  for  any  training  designer. 


The  sketch  at  the  right  outlines  the 
context  in  which  the  specification  of  design 
plans  for  the  Army  Handgun  Training  System 
takes  place.  The  total  process  begins  with 
the  generic  performance  requirements  of  the 
Design  Subsystem  of  any  training  system,  i.e., 
the  taxonomy  given  in  Figure  7  above.  The 
next  step  is  the  translation  of  those  general 
performance  taxa  into  terms  keyed  to  the 
peculiarities  of  the  training  application 
of  interest;  in  this  case.  Army  Handgun 
Training.  Next  comes  the  development  of 
concrete  plans  for  insuring  that  the  Army 
Handgun  Training  Design  Subsystem  will 
possess  its  specifically  required 
potentialities,  carry  out  its  particular 
processes,  and  deliver  the  unique  products 
required  of  it.  Sample  segments  of  such 
concrete  plans  appear  in  this  section  of 
the  report  and  in  Appendix  B.  Once  the 
plans  are  approved,  they  must  be  imple¬ 
mented.  In  the  context  of  our  particular 
example,  that  means  that  steps  must  be 
taken  to  assemble  the  people,  equipment, 
and  procedures  needed  to  develop  effective 
handgun  training  for  Army  handgun  users; 
b>  conduct  activities  appropriate  to  that 
development;  and  finally,  to  deliver  usable 
handgun  training  products.  At  the  final 
stage  of  development,  those  products  are 
collected  into  an  effective  Program  of 
Army  Handgun  Instruction  that  will  enable 
soldiers  to  use  their  handguns  to  do  the 
jobs  for  which  the  weapons  are  intended. 


E.  The  Computer-Aided  Model 
1 .  General 

The  interactive  computer  system  is  being  developed  to  aid  users  of 
the  APM  understand  and  apply  the  structure  of  the  training  system  design 
model  and  to  relate  training  goals  and  activities  to  appropriate  learner 
progress  measures.  The  system  is  being  illustrated  through  a  set  of 
PASCAL-based  programs  operating  on  complex  data  bases  using  an  Apple-11 
desktop  computer. 


It  was  recognized  early  that  the  STM  and  its  emcompassing  APM  are 
somewhat  toilsome  to  use.  It  was  foreseen  that  a  streamlining  of  the  process, 
a  reduction  in  the  magnitude  of  effort  required,  and  greater  ease  in  applying 
the  model  can  be  achieved  if  an  interactive  computer  system  is  employed.  This 
research  task  was  intended  to  determine  he  feasibility  and  resource 
implications  of  incorporating  parts  of  the  APM  (specifically,  the  STM)  onto  an 
interactive  computer  system  for  machine-aided  applications.  The  task  included 
the  examination  and  step-by-step  documentation  of  procedures  followed  in  one 
or  more  manual  applications  of  the  model.  It  then  assessed  how  each  step  can 
be  programmed  on  an  interactive  computer,  especially  those  steps  in  the  model 
application  which  necessitate  the  analyst's  review  of  taxonomies.  A  sample 
application  of  the  model  on  a  small  interactive  system  was  also  begun  in  this 
task . 

2 .  Points  of  Departure  for  This  Task 

Since  the  APM  is  intended  for  use  with  any  human-machine  system, 
and  all  system  development  or  evaluation  efforts  have  finite  and  limited 
resources  of  time,  money,  and  personnel,  it  is  necessary  to  find  ways  of 
simplifying  and  hastening  the  analytic  processes  without  undermining 
confidence  in  the  results.  If  found  to  be  too  time-consuming  or  costly,  any 
model  risks  being  abandoned  by  potential  users,  regardless  of  how  effective  it 
can  be. 

Consider  the  magnitude  of  the  process  (or  algorithm)  for  creating  a 
general  set  of  taxonomies  and  guidelines  for  applying  the  STM  to  any 
particular  human-machine  system.  There  is  no  doubt  that  the  total  number  of 
populations  that  exist  among  the  universe  of  human-machine  systems  is 
enormous.  However,  if  one's  principal  interest  is  centered  on  a  particular 
sub-universe  of  systems  (e.g.,  training  systems  for  military  weapons),  it  may 
be  possible  to  identify  a  finite,  manageable  set  of  populations  whose  members 
span  the  range  of  variables  pertinent  to  performance/effectiveness  specification 
of  measurement  of  (at  least)  most  of  the  systems  belonging  to  that 
sub-universe.  In  particular,  it  may  be  possible  to  organize  that  finite, 
manageable  set  into  subsets  corresponding  to  the  cells  formed  by  interaction  of 
the  first  two  dimensions  of  the  STM  (see  Figure  2).*  This  would  produce 

*  At  present,  the  model's  third  dimension  (the  System  Hierarchical  Structure) 
will  be  implicitly  represented,  based  on  the  extent  to  which  other  systems 
which  interact  with  the  system  of  interest  are  covered  by  the  automated 
model's  data  base. 


t 


fifteen  sets  of  system  taxonomies,  with  each  set  corresponding  to  the 
interaction  of  one  of  the  three  Descriptive  Levels  with  one  of  the  five 
Contextual  Aspects.  The  existence  of  these  sets  presumably  would  greatly 
facilitate  the  application  of  the  STM  to  any  particular  human-machine  system 
belonging  to  the  sub-universe  of  interest.  To  identify  the  total  set  of 
taxonomies  for  that  system,  one  would  search  through  each  of  the  fifteen  sets 
in  turn,  "pulling  out"  the  populations  to  which  the  system  belongs. 
Occasionally,  one  possibly  would  discover  that  the  system  also  belongs  to  a 
population  not  contained  in  the  available  sets.  That  population  could  be 
added,  so  that  the  fifteen  basic  sets  would  "grow  adaptively",  incorporating 
more  of  the  taxonomies  associated  with  more  of  the  systems  of  their 
sub-universe. 

The  compilation  and  use  of  taxonomy  sets  should  be  greatly 
facilitated  by  automated  data  processing.  In  one  of  the  trial  applications  to 
just  one  subsystem  of  the  BIFV,  the  STM  produced  several  hundred 
taxa.  The  "finite,  manageable  set"  of  taxonomies  for  the  class  of  all  weapons 
systems  easily  could  have  several  thousand  or  even  tens  of  thousands  of 
entries.  Manual  searching  and  extracting  from  such  voluminous  sets  would  be 
very  laborious  and  error-prone.  But  if  the  sets  were  maintained  in  an 
interactive  computer  system,  it  might  be  possible  to  reduce  the  labor  involved 
significantly.  Ultimately,  supporting  software  could  be  written  so  that,  given 
certain  descriptive  features  of  the  system  of  interest  as  input,  the  automated 
routine  might  take  a  "first  cut"  at  selecting  at  least  the  Objectives  Level  Taxa 
(corresponding  to  the  "what"  of  system  performance).  Of  course,  the  human 
analyst  could  always  amend  those  initial  choices  by  deleting  or  adding  taxa, 
but  at  least  his  or  her  search  time  should  be  shortened  substantially. 
Furthermore,  the  computer-based  implementation  eventually  could  include  a 
permanent  memory  of  all  of  its  past  taxonomization  applications.  This  could 
allow  keeping  track,  for  example,  of  the  frequency  with  which  each  population 
in  each  lexicon  has  been  selected  and  of  the  groupings  of  populations  that 
have  occurred  in  past  applications.  The  sequence  in  which  populations  are 
presented  to  the  human  analyst  for  his  or  her  consideration  could  be 
continually  refined  based  upon  the  observed  frequencies  and  groupings  of 
previous  selections.  Given  enough  previous  applications,  the  software  could 
become  progressively  "smarter,"  and  become  capable  of  making  steadily  more 
accurate  initial  selections  of  taxonomies  based  on  what  it  has  "learned"  in  the 
past.  Although  it  may  never  be  possible  to  dispense  entirely  with  the  human 
analyst  in  the  final  selection  of  taxonomies,  it  should  be  possible  to  eliminate 
much  of  the  work  that  presently  is  required.  It  should  also  be  relatively  easy 
to  design  the  memory  to  permit  new  entries  to  be  made,  and  even  to  delete 
old,  never-chosen  entries  as  time  goes  on. 

The  PASCAL  programming  language  was  selected  for  this  sample 
application  because  a  form  of  PASCAL  is  expected  to  be  available  in  about  1984 
for  use  at  remote  terminals  throughout  the  Department  of  Defense.  Various 
versions  of  PASCAL  are  now  available  for  use  on  such  computers  as  the  Apple 
II  and  the  DF.C-20,  both  of  which  were  potentially  available  for  this  task. 
Except  for  the  newly  announced  IBM  Personal  Computer,  no  PASCAL  compilers 
have  been  developed  for  other  IBM  computers  to  date,  as  far  as  can  be 
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determined.  Recently  published  manuals  on  Standard  PASCAL,  UCSD* 
PASCAL,  Apple  PASCAL  and  PASCAL-20  have  been  obtained  for  reference  in 
this  task.  Although  a  desktop  computer  like  the  Apple  II  would  not  have  the 
capacity  to  handle  the  model's  ultimate  information  content,  it  was  chosen  as 
useful  and  convenient  in  this  early  demonstration  stage  to  execute  a  sample 
program  which  can  later  be  expanded  and  transferred  to  a  larger  capacity 
machine. 


Selection  of  the  Sample  Application  to  be  Programmed 


The  BIFV  system  has  been  analyzed  (manually)  more  than  any  other 
with  this  STM/APM  model,  and  it  also  is  one  in  which  major  practical  interest 
lies.  Specifically,  the  most  advanced  application  of  the  model  to  date  is  with 
the  BIFV  Training  System,  an  important  entity  for  which  a  general  set  and  a 
particular  subset  of  taxonomies  have  been  identified  under  the  first  several 
model  components.  Because  of  this  experience  in  applying  the  model  and  the 
project  team's  existing  knowledge  of  the  BIFVTS,  the  basic  algorithm, 
guidelines  and  data  sets  are  fairly  well  understood  and  were  adaptable  to  a 
sample  computer  application  with  less  effort  than  if  one  were  to  select  a  system 
not  previously  examined. 


In  order  to  keep  the  scope  of  this  task  within  project  resource 
limits,  and  since  this  was  to  be  a  sample  application  only,  just  one  subsystem 
(Design)  of  the  BIFVTS  was  considered  for  the  main  computer  implementation 
(see  Figure  5  earlier  in  this  report).  Furthermore,  this  implementation  was 
carried  down  through  only  one  set  of  task  requirements  associated  with 
Curriculum  Development,  namely,  Specification  of  Learner  Testing.  Other  sets 
of  tasks  within  Curriculum  Development  are: 


o 

o 

o 

o 

o 

o 

o 

o 

o 


Specification  of  Training  Objectives 
Specification  of  Training  Content 
Specification  of  Training  Methods 

Specification  of  Texts,  Lesson  Plans  and  Reference  Sources 

Specification  of  Instructor  Requirements 

Specification  of  Facilities  Requirements 

Specification  of  Instructional  Schedules 

Secification  of  Training  Standards 

Etc. 


By  means  of  the  narrowly  defined  example,  this  task  attempted  to  illustrate  the 
feasibility  and  value  of  guided  general  conceptualizations  for  helping  to 
evaluate  any  system,  the  use  of  pre-established  taxonomies  at  general  and 
specific  levels,  the  improvement  of  data  bases  in  adaptive  computer-aided 
processes,  and  the  ability  of  the  process  to  yield  specific  implications  for 
system  definition  or  evaluation  measurement  (if  not  some  recommended 
specifications  or  effectiveness  measures  themselves). 


sity  of  California  at  San  Diego. 
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4. 


Implementation  Concepts 

The  key  notion  of  branching  down  through  sets,  to  subsets,  and  to 
sub-subsets  is  evident  from  the  first  steps  in  preparing  for  STM  application. 
Figure  21  illustrates  how  the  focus  of  evaluation  and  measurement  narrows  from 
all  human-machine  systems,  to  all  training  systems,  to  the  BIFV  Training 
System,  to  the  BIFVTS  curriculum  development  subsystem,  and  finally  to  the 
specification  of  learner  testing  within  that  subsystem. 

The  STM/APM  components  have  a  similar  nesting  structure,  such  as 
illustrated  in  Figure  7.  In  that  Figure,  taxonomies  for  a  general  design 
for  Curriculum  Development  subsystem  are  listed  for  the  first  three  contextual 
aspects  (Performance  Potentialities,  Performance  Processes,  and  Performance 
Products)  at  all  three  levels  of  system  description  (Objectives,  Functional 
Purposes,  and  Characteristics).  It  is  seen,  by  referring  to  the  numbering 
scheme,  that  the  sets  of  taxonomies  at  the  higher  levels  branch  down  to  nested 
subsets  at  the  lower  levels.  The  taxonomies  for  the  "Specification  of  Learner 
Testing"  are  a  subset  of  those  in  Figure  7.  In  operation  with  a  computer,  the 
analyst  could  be  shown  a  display  of  the  general  Design  taxonomies,  and  could 
identify  for  the  computer  those  which  apply  to  the  area  of  specific  interest 
(i.e.,  Specification  of  Learner  Testing  in  the  particular  training  system  being 
considered).  The  computer  could  then  guide  the  analyst  through  subsequent 
"menus"  and  selections  until  all  the  STM  components  have  been  applied  at  all 
system  levels,  and  implications  for  evaluation  measurement  or  system 
specification  can  be  derived.  Similar  programming  of  the  remaining  components 
of  the  overall  APM  (beyond  the  STM  portion)  could  lead  to  identification  of 
actual  measures  or  performance  requirements,  and  eventually,  perhaps,  to 
recommendations  for  applying  those  identified  measures  or  requirements. 

5 .  Computer  Programming  Implications 

Ultimately,  a  computer-aided  realization  of  the  entire  analytic  process 
is  a  major  design,  programming,  and  clerical  task.  The  sample  illustration  of 
the  computer-aided  analytic  system,  developed  using  PASCAL  on  a  small 
computer,  contains  only  a  small  fraction  of  the  information  intended  for  the  full 
system.  More  critically,  although  the  demonstration  appears  to  have  the 
capabilities  of  the  full  system,  many  of  the  capabilities  are  simulated  rather 
than  developed  as  in  the  full  system.  (For  example,  searches  may  be  done  by 
simple  linear  procedures  that  would  be  too  inefficient  for  use  in  the  full 
system. ) 


In  the  development  of  the  illustration,  however,  design  concepts  for 
the  full  system  were  considered.  Only  after  the  major  design  decisions  for  the 
full  system  have  been  made  can  a  realistic  subset  be  extracted  for  use  as  an 
illustration.  In  this  development,  the  user-computer  interactions  and  the 
user's  abilities  and  experience  are  key  concerns.  As  interactive  computers 
have  become  more  common  in  recent  years,  techniques  have  been  developed  to 
enable  more  efficient  use  of  computer  facilities  by  less  expert  users. 
Particular  concern  is  being  paid  to  relatively  naive  users--ones  who  seldom  use 
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computers  or  who  are  unfamiliar  with  the  current  system.  Techniques  such  as 
"menu  selection,"  extensive  "help"  facilities,  and  interactive  error  correction 
procedures  all  make  it  easier  for  naive  users  to  conduct  work  successfully  on 
the  computers  without  the  need  to  reference  written  support  documents  or  to 
call  upon  system  experts  for  aid.  The  best  of  these  procedures  are  sought 
and  used  as  appropriate  in  the  design  of  the  user-machine  interface  for  this 
task.  For  example,  split-screen  formats  are  considered  to  display  "menu"  data 
and  user  guidelines  or  instructions  at  the  same  time. 

The  data  structure — the  taxonomic  information  is  being  organized  to 
reflect  the  complex  interrelationships  to  be  exploited  by  the  users.  The  basic 
elements,  or  nodes  in  the  structure,  will  be  made  up  of  key  pieces  of 
information  for  the  user.  But  relating  the  nodes  to  each  other  is  the  key  to 
having  a  powerful  and  useful  data  base.  Multiple  kinds  of  links  were  explored 
and  are  implemented  as  appropriate.  For  example,  the  overall  data  base  has  a 
tree  structure  like  that  partially  shown  in  Figure  21,  with  successive  nesting 
of  subsets  to  more  and  more  specific  detail.  But,  to  be  useful  in  an 
interactive  environment,  the  data  base  must  have  other  ways  of  being  scanned. 
For  example,  one  must  be  able  to  access  all  "related"  taxa  of,  say, 
performance  processes.  And  "related,"  to  be  meaningful,  may  require  each 
node  or  cluster  of  nodes  to  be  judged  on  several  content-descriptive 
dimensions . 

To  access  the  data  structure,  the  user  must  be  able  to  ask  powerful 
questions,  to  use  general  queries  to  expand  understanding,  and  to  use 
intermediate  answers  to  guide  subsequent  questions.  This  query  system, 
dependent  upon  the  data  structure  for  its  detail,  must  be  a  helpful  and 
productive  tool  for  the  user. 

Obviously,  the  more  features  present  in  the  full  computer  system, 
the  more  flexible,  powerful,  and  ultimately  valuable  to  the  user  it  can  be.  To 
program  such  features  and  to  input  a  data  base  of  the  required  complexity  can 
be  extremely  costly,  however,  and  a  middle  ground  must  be  chosen  such  that 
the  end  product  is  both  useful  and  cost-effective.  At  this  point  in  the 
project,  it  is  intended  that  the  system  initially  be  designed  with  as  many 
features  as  possible.  Designing  is  relatively  inexpensive.  With  an 
overdesigned  starting  place,  one  can  more  knowledgeably  develop  the  actual 
implementation.  That  is,  one  can  compromise  more  effectively  knowing  the 
costs  and  benefits  of  specific  capabilities  than  by  starting  with  a  reduced 
design  in  which  the  "extra"  features  have  never  been  specified.  Moreover,  it 
may  be  that  some  desired  features  initially  judged  too  cumbersome  can 
eventually  be  put  into  the  system,  as  costs  of  some  operations  are  reduced  or 
as  more  efficient  procedures  are  devised.  Such  additions  will  be  more  easily 
implemented  if  the  original  design  included  them  than  if  they  were  never 
contemplated  in  the  initial  work. 

The  outline  in  Figure  22  is  an  ordered  list  of  items  relevant  to  the 
design  of  a  complete  computer  system.  Included  are  features  for  the  system, 
possible  capabilities  of  the  system,  key  questions  about  the  user  population 
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ill  Human -Machine 
Systems 


Others:  Atomic,  Planetary,  Animal 
Geological,  Weather,  etc. 


All  Training 
Systems 


Others:  Communication,  Transporta¬ 
tion,  Weapon,  Maintenance, 
Health  and  Welfare,  Supply, 
Production,  Navigation,  etc. 


BIFV  Training 
System 


Others:  University,  Parent- Child, 
Auto -Driving  School,  Radio 
Operator,  9mm  Handgun, 
etc. 


Design 

Subsystem 


Others:  Command,  Enabling,  Em¬ 
placement  Logistics,  Delivery 


_ 1  Others:  Training  Objectives,  Training 

Specification  of  Content,  Training  Methods, 

Learner  Testing  Information  Sources,  Instruc- 

|  tor  Requirements,  Facilities 

(Begin  Detailed  Application  of  Requirements,  Instructional 

the  STM/APM)  Schedules,  Training  Standards 


Performance  Performance  Performance 

Potentialities  Processes  _ _ Products  _ 

—  Objectives  Levell— ” Objectives  Level  Objectives, . . 
-•Functional  Pur-  r*  Functional... 

poses  Level 

—  Characteristics  Level 


Environment 


Figure  21.  Branching  Down  Through  Nested  Sets  as  a 
Basic  Concept  in  Applying  the  STM 


Utrff:  »*-g-  of  kinds  of  tacrs.  outgroups  described  in  Ur  ms  of 

A.  Bonk  or  position 
B  Silellgenee 

C.  Content  ores  experienee/treinlng 

D.  Computer  experience /training 
E  Purposes  in  mini  the  system 

F.  Experience  with  this  syitem— amount,  recency 


a  Oats  boor: 

A.  Magnitude  of  the  dots  base 
1.  Bow  much  text 

t.  Bow  many  nodes  (eg.,  taxe,  performance  mpects.  measurement  Naurs. 
I.  Bow  many  node  descriptors  MOEs/MOPs) 

4.  Bow  much  tout  storage  required 

B  Structure,  eg., 

1.  Treelike  hierarchy 

a.  Up  and  down  links 

b.  Jump  to  parallel  tranches 

].  Overall  structural  dimensions,  eg.,  matrix  dimensions: 

a-  .  Descriptive  levels 

b.  Contextual  aspects 

e.  System  Families  (Hierarchical  Blruetiee) 

t.  Mode  or  aubstruetiee  qualifiers 

a.  Attributes,  relevance  end  values  or  value  ranges 

b.  Ordered  links  (eg..  A  Mines  B,  B  precedes  C) 

e.  Mmilarity  links  (eg..  A  Is  like  S.  (A  to  b)  le  like  (C  to  E 
d.  Co-occurrence  links  (eg..  A  comes  up  whenever  B  is 
mentioned,  or  A  never  exists  alongside  B> 
a.  Bitensity  of  links,  eg.,  week  links  passed  over  in  general 
aearchcs  but  followed  In  In-depth  searches 

4.  Bode  contents 

a.  Same  or  different  in  different  puts  ef  etnieture 

b.  Barge  of  types  of  node  eontents.  if  mere  then  one 
a.  Necessary  type/ettribute  Identifiers 

I.  StaelbUity  r'  multiple  Ramification  (struct we)  aehemet 

B  Qurry  languagt/uaer  interface 

A.  Oeneral  design  features,  eg.. 

1.  Menus  for  users  to  search  to  make  branching  choice. 

5.  titerrupt  with  stack  memory— whether  ueers  can  biterrupt  to 
perform  a  new  task,  then  resume  the  prior  task 

I.  "Help"— e  general  panic  button  to  have  the  system  provide 

oxtrs  information  to  help  the  user  find  out  where  he  is  end 
Whet  he  can  do 

4.  Best  ore  .'beck  up— the  ability.  In  the  middle  of  a  complex 

search,  to  back  up  one  or  store  decision  points  and  resume 
In  another  search  direction— without  having  to  start  anew 
t.  Trace  printout— the  ability  to  arder.  at  any  point,  a  permanent 
•spy  of  acme  or  all  of  the  previous  search  results 
i.  Tutorial  subprogram,  to  train  naive  uaers  of  the  system  on 
what  the  capabilities  are  and  how  to  Interact 
T.  User  comments— e  facility  by  which  users  can  comment  on 
miything  related  to  the  system  In  messages  to  the  system 
aatnulrien— with  eventual  feedback  to  ^ocifie  soars 

B.  Batiae  of  totarac lions 

1.  Fixed-format  war  inputs  vs.  relatively  free -format 
*.  Command  defaults  to  allow  straightforward  actions  far  naive 
■ears  plus  powerful  abilities  far  sophisticated  seers 
I.  Command  prompting  to  help  wars  compose  commands 
4.  Bnflish-iikf  commands 

t.  Forgiving  of  minor  errors  bs  constructing  commands 
g.  Quick  system  response  time 


Keyed  to  strvetwe  ef  date  base 
Sequential  or  parallel  screening 
Allow  tagieally  camples  selection  ndes 


Figure  22.  Items  Relevant  to  the  Design  of  the 
Computer-aided  System 
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and  the  system  uses,  and  other  related  issues.  The  list  is  an  initial  gathering 
of  items  considered  in  the  design,  rather  than  an  outline  of  the  system  or  a 
set  of  steps  to  follow  in  the  design  process. 

Other  aspects  of  the  system,  particularly  record-keeping  and 
dynamic  modifications,  represent  a  secondary  level  of  design  which  need  only 
be  briefly  discussed  here.  The  system  must  keep  track  of  who  has  used  it 
and  for  how  long,  and  there  may  be  value  to  recording  some  session  details  as 
well.  The  system  will  permit  modification,  of  course,  but  an  important 
consideration  is  the  source  of  such  changes.  The  system  designers  can 
enlarge  or  modify  the  data  base  and  change  the  interactive  capabilities  of  the 
query  system  at  will.  The  direction  for  such  changes  may  come  from  outside, 
i.e.,  from  content  or  system  experts  who  have  determined  that  a  system 
expansion  is  in  order.  Changes  may  be  originated  by  user  comments — ones 
w'hich  report  errors,  awkwardness,  or  weaknesses  in  system  performance. 
Finally,  changes  may  occur  because  of  the  nature  of  activities  on  the  system. 
If  users  always  make  one  choice  at  a  decision  point,  for  example,  the  overt 
decision  may  be  eliminated.  If  some  choices  or  associations  are  always 
rejected,  the  data  may  be  restructured  to  eliminate  the  possibility  of  those 
links.  In  these  last  examples,  because  the  information  needed  for  system 
change  is  resident  in  the  system  activity,  it  is  theoretically  possible  to 
program  the  system  to  monitor  such  activity  and  modify  itself. 

6 .  Specific  Demonstration  of  Interactive  Computer  Capabilities 

Design  goals  for  the  ultimate  computer  system  include  power  and 
user-friendliness.  While  the  material  to  be  make  available  to  users  through 
the  computer  system  could  be  aggregated  into  printed  volumes,  the  interactive 
approach  has  several  potential  advantages.  Paramount  is  the  opportunity  for 
the  user  to  only  investigate  the  sections  of  the  full  STM  relevant  to  .a  specific 
task  and  to  be  able  to  study  —  under  the  guidance  of  the  system — all  the 
relevant  components.  For  a  computer  system  which  is  well  designecT  and 
implemented,  users  will  corne  back  to  it  frequently  and  benefit  from  the 
information  available.  The  simplified  program  to  demonstrate  the  kinds  of 
capabilities  attainable  with  a  computer-aided  version  of  the  STM  actually 
possesses  far  less  than  the  full  range  of  capabilities  ultimately  envisioned. 
However,  the  program  and  data  bases  are  designed  to  clarify  the  full  range  of 
capabilities,  through  a  combination  of  simulation  and  inference.  The  STM 
demonstration  program  is  a  partially  working  model:  virtually  all  of  the 
options,  functions,  data  bases,  etc.,  that  ultimately  will  be  available  to  a  user 
of  a  full-fledged  automated  STM  appear  to  be  present  in  the  demonstration 
version,  but  many  of  these  would  not  actually  "work"  if  the  user  were  to 
attempt  to  select,  implement,  or  access  them. 

Using  this  approach,  the  computer  system  presently  includes  four 
components  briefly  noted  below: 
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General  Design  Subsystem  database  -  a  tree-like  hierarchy  of 
curriculum  development  concepts — goals,  processes,  and 
products — capable  of  being  divided  into  subsets  based  on 
content  as  well  as  hierarchy  factors.  Each  "node"  in  the  tree 
includes  limb  information  to  related  nodes  in  the  tree, 
descriptive  information,  and  links  to  measures  useful  in 
determining  whether,  in  an  actual  training  situation,  the  node's 
goals  have  been  met. 

Measures  database  -  a  structure  of  kinds  and  examples  of 
measures  for  training  situations  to  scale  how  well  certain 
training  activities  have  met  their  goals.  The  measures  are 
ordered  in  a  simple  structure  of  more  general  and  more  specific 
examples  and,  in  this  instance,  are  linked  to  specific  points  in 
the  content  database. 

Programs  to  allow  system  maintainers  to  conveniently  add  to  or 
modify  the  data  bases. 

Programs  for  system  users  to  selectively  study  the  portions  of 
the  databases  relevant  to  their  own  activities. 

The  system  is  intended  for  sophisticated  and  for  naive  users. 
Typically,  user  activity  choices  are  displayed  on  the  screen  whenever  user 
inputs  are  required.  Abbreviations  are  permissible,  so  that  users  can  type  as 
much  or  as  little  as  they  need  for  their  own  confidence.  A  wide  range  of  help 
is  available  at  nearly  every  point  in  a  session,  and  repeated  errors  lead  to 
attempts  by  the  system  to  help  the  user  out  of  his  dilemma.  Much  of  the 
user's  activity  involves  tracing  paths  through  the  databases  to  identify  helpful 
content  nodes  and  associated  measures.  For  any  session,  the  system  retains 
the  search  route  so  that  users  can  retrace  their  steps  as  desired.  The  system 
also  allows  information  to  be  printed  at  nearly  any  point  so  that  permanent 
records  are  available  after  the  sessions. 

Figure  23  compares  the  ultimate  capabilities  desired  of  a  full-scale 
automated  STM  with  the  representations  of  those  capabilities  in  the 
demonstration  program.  Appendix  C  presents  details  on  the  program,  in  the 
form  of  a  flow  chart  with  explanatory  narrative.  The  demonstration  based  on 
the  concepts  of  Figure  23  and  Appendix  C  are  considered  to  convey  adequately 
the  ultimate  power  of  an  automated  STM,  without  having  required  a  severe 
drain  on  pmject  resources. 

Currently ,  the  initial  formulation  of  the  programs  and  initial 
exemplary  databases  are  being  completed  on  the  Apple  II.  More  detailed 
design  goals  and  descriptions  of  the  computer  system  are  presented  in 
Appendix  C. 
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THE  FULL-SCALE.  ULTIMATE 
VER5I0N  OF  THE  AUTOMATED 
STM  SHOULD: 

Contain  complete  taxonomies  for  a 
vide  variety  of  system  types.  Each 
taxonomy,  for  each  type  of  system, 
aftould  encompass  all  of  the  system's 
subsystems,  and  Vtould  cover  poten¬ 
tialities.  processes,  products, 
environments,  and  constraints  on 
the  objectives,  functional  purposes, 
and  characteristics  levels  of  system 
description. 


Provide  two  different,  but  related, 
classes  of  application,  via.,  l)  the 
■election  of  measures  of  perfor¬ 
mance  and  effectiveness,  and 
I)  the  development  of  eystem 
design  Reifications. 

Provide  measures -relevant  taxa  for 
a  wide  variety  of  aapects  of  per¬ 
formance  for  any  selected  eystem. 
ft)  particular,  if  the  wear  has  indi¬ 
cated  interest  in  training  systems." 
the  user  should  be  able  to  Rdfy 
any  of  a  large  number  of  aspects 
of  training  system  performance  for 
which  particular  aubaets  of  taxa 
have  been  identified  ft)  the  data 
base. 


Provide  a  comprehensive  listing  of 
the  issues  and  implications  for 
measurement,  relevant  to  a  speci¬ 
fied  aspect  of  performance,  that 
derive  from  each  member  of  the 
taxa  subset  associated  with  that 
aspect  of  performance. 


Provide  suitable  measure  or  meas¬ 
ures  to  address  every  iasue  and 
Implication  raised  by  every  taxon 
associated  with  a  specified  agxct 
•f  performance. 


THE  DEMONSTRATION  VERSION  OF 
THE  AUTOMATED  STM  DOES: _ 

Contain  taxonomies  only  for  two  sub¬ 
systems  of  a  generalized  training  sys¬ 
tem.  Those  two  subsystems  will  be 
curriculum  development  end  instructor 
preparation.  Further,  no  taxa  of 
environment  or  general  constraints 
will  be  included.  The  demonstration 
program  will  convey  the  Impression 
that  other  types  of  systems  ere 
available  in  the  data  base,  but  in 
reality  none  will  be  •slecu.'de. 

ftnply  that  both  applications  ere 
available,  but  in  reality  software  will 
be  prepared  only  to  support  measures 
aelection. 


Provide  messures-relevant  taxa  for  no 
more  than  two  aspects  of  curriculum' 
development  performance  (probably 
"testing  Reification”  end  Teaming 
ebjeetives  Reification")  and  for  onlv 
one  aspect  of  instructor  preparation 
performance  (probably  "conduct  of 
testing").  Other  aspects  of  perfor¬ 
mance  wO]  appear  to  have  been  iden¬ 
tified  to)  the  date  base,  but  their  taxa 
subsets  will  not  really  have  been 
defined.  Note,  however,  software  will 
be  prepared  that  will  permit  a  user  to 
define  a  taxa  sublet,  on-line,  for  a 
newly  Reified  SRct  of  performance 

Provide  aelected,  exemplary  measure¬ 
ment  iasucs-etrictly  relevant  to  "test- 
big  Reification"  performance— and 
only  for  the  objectives  level  and  func¬ 
tional  purposes  level  taxa  associated 
with  a  curriculum  development  sub¬ 
system's  performance  of  "testing  r- 
•ification."  Note,  however,  software 
will  be  prepared  that  will  permit  • 
user  to  define  measurement  iasue 
Information,  on-line,  for  any  Reified 
taxon. 

Provide  measures  for  the  issues  raised 
by  only  a  select  asmple  of  the  Use 
associated  with  Testing  Rcifiettion." 
Hots,  however,  the  ability  of  the  user 
to  define  measures,  an-iine,  wO)  be 
demonstrated. 


Figure  23.  Comparison  between  Demonstrated  and  Ultimate 
Capabilities  of  a  Computer-aided  STM 


IV.  A  PM  APPLICATION  PROCEDURES 


The  two  general  applications  of  the  Analysis  Process  Model  are  to  the 
design  of  human-machine  systems  and  to  the  measurement  of  the  performance 
and  effectiveness  of  those  systems.  The  foregoing  discussions  and  examples  of 
those  applications  in  the  training  system  context  should  have  made  one 
fundamental  point  abundantly  clear:  system  design  and  system  measurement 
deal  with  exactly  the  same  issues.  Those  issues  are  the  constituent 
requirements  of  system  performance.  The  designer  and  the  evaluator  are  (or 
should  be)  interested  strictly  in  the  capabilities  (potential  performance)  their 
system  is  supposed  to  have;  the  activities  (process  performance)  it  is 
supposed  to  carry  out;  and,  the  goods  and  services  (product  performance)  it 
is  supposed  to  deliver.  The  designer's  job  is  to  see  to  it  that  the  system  does 
in  fact  meet  all  of  its  requirements  for  potential,  process,  and  product 
performance.  The  evaluator's  job  is  to  check  on  how  well  the  designer's  job  is 
done . 

Sometimes  a  system  designer  may  achieve  somewhat  more  than  was 
required:  the  system  might  not  only  deliver  all  required  products,  for 

example,  but  also  some  additional  useful  goods  or  services  that  weren't  among 
those  essential  to  meet  the  needs  of  the  system's  sponsor.  Designers  expect 
those  kind  of  achievements  to  be  applauded  and  rewarded.  Evaluators, 
however,  have  to  take  a  long,  skeptical  look  at  such  design  "extras."  The 
evaluator  has  to  ask,  how  much  is  this  additional  item  going  to  cost  the 
sponsor?  Could  any  economies  in  time,  equipment,  funds,  or  other  resources 
be  realized  if  we  dispensed  with  this  "extra"?  Most  importantly,  does  the 
delivery  of  this  "extra"  in  any  way  impair  the  system's  ability  to  have 
available  its  essential  capabilities,  carry  out  its  essential  activities,  or  deliver 
its  essential  goods  and  services?  Depending  on  how  these  kinds  of  questions 
are  answered,  the  designer's  expected  applause  might  materialize  as  boos. 
The  evaluator  might  report  that,  far  from  deserving  additional  credit,  the 
"extra"  good  or  service  might  really  be  a  system  defect,  that  absolutely  must 
be  removed.  The  evaluator  is  the  designer's  advocatus  diaboli:  the  one  who 
has  to  insist  on  rigorous  proof  that  the  designer's  work  has  met  all  applicable 
requirements.  "Extras"  are  nice,  but  only  after  all  of  the  essentials  are  met.* 

Sometimes,  too,  the  performance-oriented  shoe  is  on  the  other  foot.  It  is 
far  too  common  that  evaluators  find  fault  with  a  system  designer's  work  that 
really  satisfies  the  performance  requirements  very  well.  Such  invalid 
objections  arise  from  the  evaluators'  preconceived  ideas  about  how  the  system 
should  achieve  its  requirements ,  instead  of  from  the  basic  question  of  whether 
the  system  performs  as  it  should.  A  case  from  the  early  days  of  World  War  II 
is  illustrative: 

^Design  Ire>:trasn  that  truly  do  not  impair  achievement  of  real  performance 
requirements  in  any  way  probably  are  as  rare  as  the  free  lunch. 
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In  order  to  maintain  her  essential  supply  of  materials 
from  abroad,  Britain  equipped  much  of  her  merchant 
fleet  with  anti-aircraft  guns.  Those  guns  were  in 
short  supply,  and  were  much  in  demand  by  the  military 
for  installation  in  shore  batteries  and  on  men-of-war. 

The  Royal  Navy  conducted  an  "evaluation"  of  the 
effectiveness  of  the  merchantmen's  guns  which  showed 
a  very  small  number  of  German  aircraft  had  been 
brought  down  by  the  merchant  fleet.  The  ratio 
of  a-a  gun  shots  to  aircraft  "kills"  was  vanish¬ 
ingly  small.  On  the  strength  of  that  study,  the 
Navy  argued  for  the  removal  of  guns  from  the 
merchant  fleet,  so  that  the  weapons  could  be 
employed  more  "effectively"  by  the  "professionals." 

That  argument  nearly  carried  the  day,  until  someone 
pointed  out  that  the  guns  weren't  there  to  kill 
German  aircraft,  but  rather  to  keep  British  ships 
afloat.  In  fact,  the  a-a  guns  were  highly  effec¬ 
tive,  in  that  a  much  smaller  percentage  of  armed 
merchantmen  were  sunk.  The  sailors  used  their 
firepower  to  keep  the  aircraft  at  bay,  and  that 
was  all  that  they  had  to  do  to  achieve  their 
performance  requirement.* 

The  designer,  then,  must  stand  ready  to  defend  the  system  and  its 
specifications  strictly  on  the  issue  of  its  performance.  The  designer  has  the 
right,  and  the  obligation,  to  insist  that  any  criticism  of  his  or  her  work  be 
shown  to  stem  from  a  real  performance  deficiency.  Otherwise,  good  work  may 
be  needlessly  corrupted. 

System  design  and  system  measurement  thus  really  are  two  different 
aspects  or  views  of  precisely  the  same  thing.  They  are  more  closely  related 
than  are  two  sides  of  the  same  coin,  for  they  interact  constantly.  At  the 
outset  of  a  system  development  project,  a  designer  takes  a  set  of  performance 
requirements,  identifies  the  attributes  of  people,  procedures,  and  equipment 
needed  to  meet  those  requirements,  and  develops  a  preliminary  plan  for 
securing  those  attributes  in  the  system  to  be  built.  The  designer's  work  at 
this  conceptual  stage  is  oriented  heavily  toward  the  performance  potentialities. 
The  plan  of  course  addresses  how  the  processes  will  be  carried  out,  and  what 
the  products  will  look  like.  But  the  plan's  basic  purpose  is  to  demonstrate 
that  the  system  will  have  what  it  needs  (in  terms  of  abilities)  to  carry  out  the 
processes  and  deliver  the  products. 


*Given  that  one  of  our  sample  applications  of  the  APM  is  to  Army  handgun 
training,  it  is  relevant  to  note  that  a  soldier's  handg  ...  .3  very  closely 
analogous  to  the  merchant  ship's  a-a  gun.  In  each  case,  the  basic  purpose  is 
to  defend  the  user,  not  necessarily  to  kill  the  enemy.  Handgun  training  thus 
logically  should  emphasize  defensive  operations.  Unfortunately,  the  training 
often  boils  down  to  target  practice. 
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Meanwhile,  the  evaluator  examines  those  same  performance  requirements 
and  their  associated  system  attributes,  and  forms  a  set  of  measures  to  be  used 
to  determine  whether  the  system  has  those  attributes  and  meets  those 
requirements.  The  evaluator  applies  those  measures  to  the  designer's  plan. 
The  evaluator's  interest,  too,  is  at  this  time  principally  on  the  requirements 
and  attributes  of  potentiality .  Will  the  system  have  what  it  takes  to  carry  out 
the  processes  and  deliver  the  products  actually  required?  Is  the  plan  sound 
and  workable?  This  is  the  proposal  stage  of  design  and  evaluation.  The 
designer  says  "here  is  how  I  intend  to  go  about  building  the  system  to  meet 
its  performance  requirements."  The  evaluator  responds  "here  are  the  specific 
performance  defects  that  I  have  found  in  your  plan."  The  designer  should 
have  a  chance  to  rebut,  to  challenge  whether  the  evaluator's  conclusions  truly 
are  based  on  the  performance  requirements.  Once  the  evaluative  conclusions 
are  validated,  the  designer  modifies  the  plan  to  remove  the  performance 
defects,  and  begins  to  build  the  "real"  system.  This  means  that  attention 
shifts  from  potential  ("on  paper")  performance  to  actual  process  performance. 
As  the  designer  implements  the  processes  the  evaluator  again  applies  the 
measures  to  determine  whether  the  requirements  are  being  met.  At  this  stage, 
the  evaluator  might  well  uncover  some  defects  in  potentiality  that  escaped 
notice  during  the  proposal  evaluation.  To  the  extent  that  some  early  products 
begin  to  emerge,  the  evaluator  might  also  find  some  product  defects.  But  the 
focus  here  is  on  process  performance.  Is  the  system  conducting  the  activities 
that  should  be  conducted?  Is  the  plan  being  followed?  Are  the  processes 
being  adapted  to  meet  the  needs  emerging  in  the  real  world  of  system 
development?  The  evaluator  continually  monitors  the  designer's  work  at  this 
stage,  from  time  to  time  reporting  "what  you  are  doing  there  isn't  quite  what 
is  needed."  Again,  the  designer  may  challenge  the  evaluator's  observations, 
to  insure  that  they  address  real  performance  defects.  Upon  conceding  that 
the  defects  are  real,  the  designer  modifies  the  processes  (and  takes  steps  to 
acquire  any  missing  potentialities  needed  to  implement  the  processes  properly) 
to  correct  those  defects.  Work  continues,  progress  is  made,  always  under  the 
watchful  eyes  of  the  evaluator.  At  some  point,  the  system  begins  to  deliver 
its  products.  Once  again,  the  evaluator  applies  the  measures.  Here,  too,  it 
is  possible  that  previously  unnoticed  defects  in  potential  or  process 
performance:  may  surface.  The  accent,  however,  is  now  on  how  well  the 
delivered  goods  and  services  meet  their  requirements  for  acceptance.  The 
designer  says  "here  are  the  goods  you  wanted."  The  evaluator  might  reply 
"these  goods  aren't  good  enough,"  and  state  exactly  how  they  are  deficient. 
Provided  that  the  evaluator's  dissatisfaction  is  based  on  valid  product 
performance  requirements,  it  will  be  incumbent  upon  the  designer  to  "make 
good"  the  goods.  The  designer  might  be  able  to  do  that  simply  by  repeating 
the  production  process  somewhat  more  carefully.  However,  it  might  also  be 
necessary  to  re- design  the  processes,  or  even  to  re-design  the  system's 
potentialities,  in  order  to  produce  the  proper  products. 

What  we  have  sketched  on  the  previous  few  pages  of  course  is  a  general 
outline  of  an  interactive  design /measurement  process  for  human-machine 
systems:  in  other  words,  a  general  procedure  for  applying  the  APM.  It 

should  be  evident  that  design  and  measurement  are  indispensible  activities 
within  this  Analysis  Process.  Further,  they  are  living  activities,  born  with 


the  system  concept,  growing  with  it  throughout  its  construction,  and  remaining 
with  it  throughout  the  system's  active  life.  Every  design  step  demands  a 
corresponding  measurement  activity.  Every  measurement  finding  demands  a 
design  response.* 

Figure  24  depicts  the  general  procedures  for  APM  application  in  the 
Training  Systems  context.  As  shown,  the  responsibility  for  designing  and 
measuring  a  training  system  is  shared  by  the  suprasystem  sponsoring  the 
development  and  operation  of  the  training  system;  by  the  Command  Subsystem; 
and  by  each  of  the  other  training  subsystems.  Training  system  design 
activities  take  place  at  two  levels:  at  each  of  the  operating  subsystems,  and 
at  the  total  training  system  level;  Command  performs  the  design  function 
commonly  known  as  "system  integration"  in  support  of  total  system  design. 
Training  system  evaluation  activities  take  place  at  all  three  levels:  each 
subsystem  informally  measures  its  own  performance  in  designing  its  approach 
to  its  performance  requirements,  and  in  meeting  those  requirements;  Command 
formally  evaluates  each  of  the  individual  subsystem's  performance,  and 
informally  evaluates  total  training  system  effectiveness;  the  sponsoring 
suprasystem  formally  evaluates  the  effectiveness  of  the  total  training  system. 
In  carrying  out  these  activities,  all  three  levels  participate  in  applying  the 
APM. 

Details  on  the  APM  application  procedures  are  given  below,  in  reference 
to  each  functional  block  in  Figure  24. 

Block  1:  Sponsor  identifies  the  basic  needs  for  which  training  is  to  be 
developed  and  delivered. 

Following  the  definition  used  in  Interservice  Procedures  for 
Instructional  Systems  Development  (Branson  et  al.  ,  1975),  a  training 
need  is  Ira  measurable  discrepancy  between  "tTie  actual  world  as  it 
exists  now  and  the  world  as  it  ought  to  be."  Such  discrepancy  of 
course  is  perceived  as  a  dearth  of  people  qualified  to  perform  some 
job  that  the  sponsor  wants  performed.  This  could  arise  because  the 
job  is  something  brand  new:  for  example,  BIFVs  are  going  to  be 
deployed  and  there  is  no  one  who  has  ever  manned  a  BIFV. 
Alternatively,  it  could  arise  because  the  sponsor  is  unsatisfied  with 
the  way  in  which  an  existing  job  is  being  performed:  for  example, 
combat  studies  may  show  that  soldiers  are  woefully  unsuccessful  in 
defending  themselves  with  their  handguns.  A  discrepancy  could  also 
stem  from  a  combination  of  these  circumstances:  for  example,  the 
Army  may  be  about  to  introduce  a  new  handgun  to  replace  its 
current  .45  and. 38  calibre  models,  and  may  be  dissatisfied  with  the 
way  in  which  soldiers  were  trained  to  use  the  older  weapons.  This 
could  lead  to  a  decision  like  the  following:  "As  long  as  we're  going 

*In  many  cases,  the  appropriate  design  response  to  a  given  measurement 
finding  is  "don't  cl.  mge  anything!" 


to  have  to  develop  new  handgun  training  anyway,  let's  do  it  right 
this  time  and  try  to  teach  solders  to  use  the  weapon  as  it  actually 
needs  to  be  used  in  combat." 

It  is  up  to  the  sponsor  to  define  the  training  needs,  and  thus  to 
establish  the  scope  of  APM  application.  These  needs  usually  aren't, 
and  need  not  be,  expressed  in  behavioral  terms:  extraction  of 
required  behaviors  from  the  identified  needs  is  a  subsequent 
responsibility  of  the  Training  Design  subsystem.  But  the  expressed 
needs  must  totally  encompass  the  job  that  the  sponsor  wants 
performed,  so  that  the  training  system  designers  can  do  their  work 
in  the  appropriate  ballpark.  Along  with  the  needs,  it  is  also  up  to 
the  sponsor  to  provide  or  point  the  way  to  all  available  information 
concerning  the  job  to  be  trained. 

Block  2:  Command  assesses  the  relevance  of  the  general  training  system 
performance  requirements  to  the  identified  training  needs. 

The  general  performance  requirements  for  the  Training  Design 
subsystem  were  given  in  Figure  7,  while  Figure  8  presented  the 
general  requirements  for  the  Training  Enabling  Subsystem.  Similar 
general  performance  requirements  exist  for  the  Emplacement, 
Logistics,  and  Delivery  Subsystems  as  well  as  for  the  Command 
Subsystem  itself.  Depending  upon  the  particular  training  needs  to 
be  satisfied,  it  might  be  that  some  of  those  general  performance  taxa 
do  not  apply  to  the  Training  System  in  question.  The  sponsor 
might,  for  example,  have  already  performed  a  job  analysis  with 
which  the  sponsor  is  satisfied  (in  effect  usurping  a  portion  of  the 
performance  requirements  and  responsibilities  of  the  Design 
Subsystem).  As  another  example,  if  the  job  to  be  trained  is  truly 
brand  new  and  radically  different  from  all  previous  jobs,  the  Design 
Subsystem  wouldn't  need  to  analyze  existing  instruction  for  that  job 
(none  would  exist).  Thus,  one  of  the  initial  responsibilities  of  the 
Command  Subsystem  would  be  to  determine  exactly  which  of  the 
general  training  system  performance  requirements  do  apply  to  the 
application  at  hand. 

Block  3:  Sponsor  selects  measures  of  training  system  effectiveness 
relevant  to  the  identified  needs. 

The  sponsoring  suprasystem  is  interested  in  "bottom  line"  results: 
it  needs  to  determine,  ultimately,  whether  its  identified  training 
needs  are  met.  From  the  sponsor's  point  of  view,  the  training 
system  will  be  effective  if  and  only  if  it  produces  the  qualified 
people  needed  to  do  the  job  that  the  sponsor  wants  done.  The 
measures  of  training  system  effectiveness  thus  are  job  performance 
measures.  The  sponsor  needs  to  know:  will  the  learners  who 
emerge  from  the  training  system  be  able  to  perform  the  job  the  way 
we  want  it  to  be  performed?  For  example,  will  the  soldiers  who 
receive  handgun  training  actually  be  able  to  defend  themselves 


successfully  in  close-quarters  combat  with  enemy  soldiers?  Will  the 
soldiers  who  are  trained  to  be  BIFV  gunners  actually  be  able  to 
operate  the  weapons  to  engage  and  destroy  enemy  targets?  If  these 
kinds  of  questions  are  not  answered  affirmatively,  the  sponsor  will 
conclude  that  the  training  system  is  ineffective,  because  it  fails  to 
satisfy  the  sponsor's  training  needs.  It  will  not  impress  the  sponsor 
if  the  training  administrator  were  to  argue  that  "Okay,  so  our 
learners  can't  really  perform  your  job;  but  you  have  to  admit  that 
we  built  some  really  superb  classrooms  and  prepared  some  very 
attractively  packaged  lesson  plans,  and  our  cafeteria  turned  out 
some  very  tasty  meals.  Doesn't  that  count  for  something?"  In  fact, 
it  counts  for  just  about  nothing:  about  as  much  as  it  counts  to  a 
bereaved  widow  to  be  informed  that  her  late  husband's  operation  was 
"technically,  a  success." 

The  sponsor  will  derive  direct  measures  of  training  system 
effectiveness  by  applying  the  APM  to  the  operating  system  for  which 
training  is  to  be  developed.  For  example,  the  APM  would  be  applied 
to  the  BIFV  Carrier  Team  Subsystem  to  determine  its  performance 
requirements,  from  which  in  turn  gunner  performance  requirements 
and  measures  of  gunner  performance  would  be  derived.  However, 
the  sponsor  will  also  need  indirect  measures  of  training  system 
effectiveness,  i.e.,  measures  that  will  disclose  whether  the  planning 
and  development  of  the  training  system  is  on  the  right  track.  Those 
indirect  measures  will  be  derived  from  the  training  system 
performance  requirements  submitted  for  review  and  approval  by  the 
Command  Subsystem.  That  derivation  is  depicted  in  Figure  24  as 
the  dashed  line  connecting  blocks  2  and  3. 

Block  4:  Command  translates  the  relevant  generic  training  system 
performance  requirements  into  expressions  specific  to 
the  identified  training  needs. 

In  the  trial  application  to  the  U.S.  Army  Handgun  Training  System, 
the  authors  learned  that  the  general  Design  Subsystem  performance 
taxa  (as  presented  in  Figure  7)  did  not  lend  themselves  as  readily  to 
handgun  training  specifications  as  one  would  wish.  The  problem  was 
not  that  the  taxa  weren't  relevant  to  handgun  training.  Rather,  the 
general  expressions  simply  weren't  tailored  to  the  nuances  of  that 
particular  training  enterprise:  they  didn't  explicitly  impart  a 
"handgun  flavor"  to  the  performance  requirements.  It  was 
determined,  therefore,  that  an  intermediate  step  was  needed  to 
bridge  the  gap  between  generic  performance  requirements  and  a 
system-specific  plan  for  meeting  those  requirements.  The 
intermediate  step  was  to  translate  the  general  requirements  into 
expressions  tailored  to  the  particular  system  of  interest  (in  our 
case.  Army  9mm  Handgun  Training).  Samples  of  the  "tailored"  taxa 
were  given  in  Figure  19,  above,  and  in  Appendix  B.  One 
comparison  is  reproduced  here: 


Corresponding  Handgun 
Training  taxon 

"Given  a  list  of  Army  handgun 
users  and  uses,  the  Design  Sub¬ 
system  must  be  able  to  define  the 
scope  of  learning  needed  for  each 
user/use  in  terms  of  the 
knowledge,  skills,  and  attitudes 
required.  " 

This  translation  produced  much  additional  verbiage,  but  it  was 
helpful  verbiage:  specific  details  to  which  design  specifications 
could  easily  be  written.  Hence,  this  "tailoring"  of  generic  taxa  to 
specific  training  needs  is  felt  to  be  an  essential  step  in  the  APM 
Application  Procedures. 

Block  5 :  Subsystem  identifies  its  attributes  relevant  to  its  needs- 
specific  performance  requirements. 

Upon  receiving  (from  Command)  its  own,  tailored  set  of  performance 
requirements,  each  training  subsystem  takes  the  first  step  toward 
planning  for  satisfying  those  requirements.  Of  course,  the  various 
subsystems  do  not  do  this  simultaneously,  but  rather  in  a  logical 
sequence:  much  of  the  Design  Subsystem's  inital  work  must  be 

completed,  for  example,  before  the  Enabling  Subsystem's  performance 
requirements  can  be  properly  tailored.  However,  at  the  proper  time 
each  subsystem  applies  the  APM  to  take  that  first  step,  which 
consists  of  identifying  the  attributes  or  features  of  the  people, 
equipment,  and  procedures  it  must  bring  to  bear  to  do  the  job. 
This  is  a  subsystem-level  analysis  of  the  performance  requirements 
to  identify  how  those  requiements  impact  on  the  subsystem's  basic 
components . 

Block  6:  Subsystem  develops  preliminary  plans  for  securing  the 
required  attributes. 

These  preliminary  plans  are  the  first  output  of  the  training  design 
process.  In  the  plans,  each  subsystem  states  the  attributes  it  needs 
to  achieve  its  assigned  performance  requirements;  documents  proof  of 
its  possession  of  certain  of  those  attributes;  identifies  those 
attributes  that  it  does  not  now  possess  and  documents  what  it  feels 
will  be  workable  means  of  acquiring  those  missing  attributes;  and 
states  the  resources  and  other  support  it  must  have  to  implement 
those  means.  The  Subsystem  is  aware  that  Command  will  carefully 
review  and  evaluate  the  submitted  plans.  Ideally,  therefore,  the 
Subsystem  attempts  to  "anticipate"  the  evaluative  measures  that 
Command  will  apply  to  the  plans,  and  writes  the  plans  in  a  way  that 
will  produce  "high  scores"  on  those  measures.  That  is,  the 
Subsystem  informally  evaluates  its  own  plans  (and  measures  its  own 


Generic  Design 
Subsystem  taxon 

"The  Subsystem  must  be 
able  to  define  the  total 
scope  of  learning." 
(Taxon  1.1.1) 
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potential  performance)  before  submitting  the  plans  to  Command. 
This  "first  tier"  of  evaluation  gives  a  valuable  boost  toward  an 
effective  design  specification  provided  that  both  the  Subsystem  and 
Command  have  applied  the  APM  properly  to  develop  valid  measures  of 
Subsystem  performance. 

Block  7:  Command  selects  relevant  measures  of  Subsystem  performance. 

This  is  the  formal  application  (by  Command)  of  the  APM  to  develop 
the  valid  measures  of  Subsystem  performance  discussed  above.  It  is 
this  set  of  measures  that  the  operating  Subsystem  attempts  to 
"anticipate."  In  practice,  of  course,  it  is  completely  reasonable  that 
Command  may  inform  the  operating  Subsystem  of  the  measures  that 
will  be  applied  to  the  Subsystem's  plans.  Command,  after  all, 
desires  that  measurably  good  plans  emerge  from  the  Subsystem. 

Block  8:  Command  evaluates  Subsystem  plans  and  identifies  performance 
deficiencies . 

This  starts  the  first  wave  of  formal  evaluation  in  the  APM  application 
sequence.  Command  applies  its  measures  to  the  plans,  producing 
concrete,  quantitative  ratings  of  their  "goodness."  The  emphasis 
naturally  is  on  the  degree  to  which  the  plans  demonstrate 
achievement  of  the  potentialities  of  performance,  although  such 
measures  of  process  and  product  as  can  be  applied  at  this  planning 
stage  will  also  be  taken.  Command  reports  its  findings  back  to  the 
operating  subsystem,  pointing  out  specific  deficiencies  that  must  be 
corrected.  These  evaluative  findings  demand  a  design  specification 
response. 

Block  9:  Subsystem  modifies  its  plan  to  correct  the  deficiencies. 

This  is  the  design  specification  response  to  the  first  wave  of 
evaluation.  Command  has  informed  the  Subsystem  that  specific 
deficiencies  exist  in  the  plan.  Those  deficiencies  include  particular 
Subsystem  attributes  that  are  essential  for  adequate  performance, 
and  which  are  missing  or  inadequately  demonstrated  in  the  plan. 
The  deficiencies  may  also  include  some  inessential  or  even  detracting 
attributes  of  the  Subsystem  that  have  been  built  into  the  plans. 
The  operating  Subsystem  modifies  its  plan  to  remove  the  attribute 
deficiencies,  i.e.,  modifies  the  proof  of  its  fitness  for  the  particular 
training  application. 

Block  10:  Command  assembles  the  Subsystem  plans  into  an  initial 
design  specification  for  the  total  training  system. 

At  this  stage,  Command  exercises  its  overall  system  management 
responsibility  and  integrates  the  various  plans  into  one 
comprehensive  proposal  for  system  design.  This  is  an  important 
responsibility.  The  integrated  plan  must  demonstrate  not  only  that 
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each  subsystem  can  do  its  proper  work,  but  also  that  all  their  work 
will  "fit  together"  well.  A  particularly  significant  aspect  of  the 
integrated  plan  is  its  ability  to  prove  that  Command  can  adequately 
discharge  its  management  responsibility  on  a  continuing  basis. 

Block  11:  Sponsor  evaluates  the  initial  total  system  design 

specification  and  identifies  deficiencies  in  training 
system  effectiveness. 

This  is  the  higher  level  equivalent  of  Block  8.  It  completes  the  first 
wave  of  formal  evaluation.  The  emphasis  remains  on  performance 
potentialities.  Here,  however,  the  focus  is  on  the  integrated 
training  system's  potentialities.  This  reflects  the  Sponsor's  abiding 
interest  in  "bottom  line"  training  results.  In  this  present  case,  the 
interest  is  on  the  potential  for  satisfying  the  Sponsor's  training 
needs.  The  Sponsor  doesn't  much  care  whether  a  defective 
potentiality  traces  back  to  the  Design  Subsystem,  or  the  Enabling 
Subsystem,  or  whatever.  Sponsor  has  "hired"  Command  to  perform 
that  level  of  diagnosis.  Sponsor  simply  documents  total  system 
deficiencies,  and  passes  these  back  to  Command  with  an  order  to 
correct  them. 

Block  12:  Command  translates  identified  system  effectiveness 

deficiencies  into  subsystem  performance  deficiencies. 

Command  performs  the  detailed  diagnosis  for  which  it  was  "hired," 
and  reports  its  findings  back  to  the  operating  subsystems.  This 
forces  another  (final)  design  specification  response. 

Block  13:  Subsystem  (again)  revises  its  plan  to  correct  its 
performance  deficiencies. 

This  is  the  "final"  design  specification  response  to  the  first  wave  of 
evaluation.  Of  course,  this  response  will  be  evaluated  by  Command 
and  subsequently  by  the  Sponsor  (after  plan  integration).  It  is 
possible  that  those  evaluations  could  lead  to  further  revisions, 
further  evaluations,  more  revisions,  etc.  Fairly  soon,  however, 
Sponsor  would  be  expected  to  lose  patience  and  start  to  look  for 
another  training  team  to  satisfy  its  needs. 

Block  14:  Subsystem  implements  its  performance  processes  in 
accordance  with  its  plan. 

Assuming  that  Command  finally  approves  the  Subsystem's  plans,  and 
Sponsor  finally  approves  the  integrated  plan,  the  plans  finally  go 
into  effect.  The  focus  of  Subsystem  performance  shifts  from 
potentialities  to  processes. 


Block  15:  Command  evaluates  Subsystem  processes  and  identifies 
performance  deficiencies. 

This  starts  the  second  wave  of  formal  evaluation.  Just  as  in 
Block  8,  Command  applies  its  previously  selected  measures  (Block  7) 
to  the  Subsystem's  performance.  But  now,  the  principal  focus  is  on 
process  performance.  The  Subsystem  has  moved  from  the  "on 
paper"  design  specification  stage  to  the  active  design  implementation 
stage.  Evaluation  similarly  moves  away  from  the  activities-as- 
planned  to  the  activities-as-implemented .  Any  deficiencies  noted  in 
that  implementation  are  reported  back  to  the  operating  Subsystem. 

Block  16:  Subsystem  revises  its  processes  to  correct  the  deficiencies. 

This  is  the  design  implementation  response  to  Command's  second 
wave  evaluative  findings.  Command  has  pointed  out  that  the 
Subsystem's  activities  are  deficient:  it  may  be  doing  some  things 
that  it  shouldn't,  it  may  be  failing  to  do  some  things  that  it  should 
do,  or  both.  The  Subsystem  responds  by  changing  what  it  does. 
Note  that  some  of  these  process  deficiencies  may  arise  because  the 
Subsystem  has  deviated  from  its  plans:  it  may  be  doing  some  things 
differently  from  the  way  it  proposed  to  do  them.  But  note,  too, 
that  some  process  deficiencies  may  arise  precisely  because  the 
Subsystem  adheres  to  the  plan,  refusing  to  change  with  changing 
conditions.  The  plan,  any  plan,  rests  on  certain  assumptions  about 
how  the  work  will  unfold.  No  plan  is  foolproof.  The  wise  system 
designer  will  not  abandon  the  plan  at  the  earliest  opportunity,  but 
neither  will  he  or  she  insist  blindly  on  sticking  to  it  when  it  becomes 
clear  that  things  aren't  working  out  as  they  were  expected  to. 
Process  evaluation  is  not  simply  a  matter  of  checking  up  on 
deviations  from  planned  performance,  although  that  is  one  facet  of 
concern.  Rather,  true  process  evaluation  is  the  determination  of 
whether  the  system  is  doing  what  it  should,  given  the  conditions  of 
the  real  world.  If  the  evaluator's  process  measures  are  based  on 
the  true  process  performance  requirements,  he  or  she  will  uncover 
the  real  process  defects,  and  not  simply  disparities  that  may  or  may 
not  be  bad. 

Block  17:  Command  controls  the  integration  of  the  various  Subsystem 

processes  into  a  total  training  system  process  implementation. 

This  is  another  stage  at  which  Command  executes  system 
management.  Now,  Command's  responsibility  is  to  see  to  it  that  the 
work-as-c,irried-out  by  the  individual  Subsystems  merges  smoothly  as 
it  should.  Previously,  the  concern  was  with  the  fitting  together  of 
the  work-as-planned. 
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Block  18:  Sponsor  evaluates  total  system  process  implementation  and 
identifies  deficiencies  in  training  system  effectiveness. 

This  completes  the  second  wave  of  formal  evaluation.  Now,  the 
Sponsor  looks  at  what  the  total  training  system  actually  is  doing, 
and  asks  whether  that  promises  progress  toward  satisfaction  af  ‘he 
training  needs.  Any  system  activities  that  are  either  a  hindi  '.e  to 
realizing  those  needs  or  merely  superfluous  are  pointed  out  to 
Command  as  process  defects. 

Block  19:  Command  translates  identified  system  effectiveness  deficiencies 
into  Subsystem  performance  deficiencies. 

This  is  identical,  in  word  and  spirit,  to  Block  12,  although  the 
deficiencies  with  which  Command  now  deals  are  likely  to  be  process- 
rather  than  potentiality-based. 

Block  20:  Subsystem  revises  its  processes  and  its  plans,  as  needed,  to 
correct  its  performance  deficiencies. 

This  is  another  design  implementation  response  to  a  set  of  evaluative 
findings.  Of  course,  changes  in  processes  that  are  underway  may 
necessitate  changes  in  processes  planned  for  subsequent 
implementation.  Thus,  design  specification  response  might  also  be 
needed . 

Block  21:  Subsystem  delivers  its  performance  products  in  accordance 
with  its  plan. 

After  all  parties  have  approved  the  revisions  to  the  processes,  the 
work  picks  up  again.  Eventually,  products  emerge.  The  focus  has 
now  shifted  from  design  specification  through  design  implementation 
to  design  fruition. 

Block  22:  Command  evaluates  Subsystem  products  and  identifies 
performance  deficiencies. 

Now  the  third  wave  of  formal  evaluation  begins.  The  Subsystem 
performance  measures  once  more  are  applied,  this  time  with  special 
emphasis  on  product  measures.  Command  now  has  something  to  say 
directlv  about  the  goodness  of  the  Subsystem's  goods. 

Block  23:  Subsystem  raises  its  products  to  correct  the  deficiencies. 

This  is  the  design  fruition  response  to  the  third  wave  of  evaluation. 
To  the  extent  that  the  Subsystem's  goods  are  deficient,  the 
Subsystem  must  "make  good"  the  goods.  Sometimes  this  merely 
requires  placing  a  few  "final  touches"  on  the  delivered  goods:  a  bit 
of  polishing  here  and  there,  a  few  more  dabs  of  paint,  correction  of 
some  typographical  errors,  deliverance  of  the  service  with  a  smile 
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rather  than  a  frown,  etc.  Other  times  the  "making  good"  requires 
scrapping  the  submitted  product  and  completely  replicating  the 
process  that  produced  it,  this  time  with  much  greater  care.  When 
that  sort  of  situation  develops,  the  blame  often  can  be  partially 
placed  in  a  defective  process  evaluation:  if  the  original  process  was 
performed  so  poorly  that  the  product  had  to  be  scrapped,  the 
evaluator  probably  should  have  discovered  that  fact  before  the 
product  emerged.  At  still  other  times,  the  only  way  to  11  make  good" 
on  the  product  is  to  scrap  the  processes  of  production  and  devise 
new,  more  effective  products,  that  is,  a  fundamental  redesign  of  the 
system  may  be  needed.  All  evaluators  know  that  this  kind  of 
situation  does  sometimes  arise.  But  they  also  know  that,  when  it 
does,  there  often  are  defective  evaluations  of  potentiality  and 
process  buried  somewhere  in  the  system’s  history. 

Regardless  of  whatever  it  takes,  any  defective  goods  need  to  be 
"made  good." 

Block  24:  Command  controls  the  integration  of  the  various  Subsystem 
products  into  a  set  of  total  training  system  products. 

Now  it  is  the  work-as-delivered  that  must  be  made  to  "fit  together." 
Conflicts  and  inconsistencies  between  the  interacting  products  of  two 
or  more  subsystems  must  be  removed,  so  that  an  acceptable  total 
package  may  be  delivered  to  the  Sponsor. 

Block  25:  Sponsor  evaluates  total  system  products  and  identifies 
deficiencies  in  training  system  effectiveness. 

This  completes  the  "final"  wave  of  formal  evaluation.  Here,  the 
Sponsor  can  come  to  grips  with  the  bottom  line  of  all  bottom  lines: 
Does  this  now-fully-assembled  training  system  actually  satisfy  the 
Sponsor's  training  needs?  To  the  extent  that  any  needs  remain 
unsatisfied,  training  product  defects  will  have  been  found. 

Block  26:  Command  translates  identified  system  effectiveness  deficiencies 
into  Subsystem  performance  deficiencies. 

Just  as  in  Blocks  12  and  19,  Command  performs  the  detailed 
diagnosis  to  determine  how  the  defective  work  can  best  be  corrected. 
Now,  of  course,  it  is  diagnosis  of  defective  products  that  claims  most 
of  Command's  attention. 

Block  27:  Subsystem  revises  its  products,  processes,  and  plans,  as 
needed,  to  correct  its  performance  deficiencies. 

This  is  another  design  fruition  response  to  a  set  of  evaluative 
findings.  Depending  upon  exactly  what  is  needed  to  "make  good"  its 
goods,  the  Subsystem  may  also  have  to  make  a  design  implementation 
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response  (revise  its  processes),  or  even  a  design  specification 
response  (revise  its  fundamental  plans  and  capabilities). 

Blocks  28,  29  and  30:  Suprasystem,  system  of  interest,  and  other 

training  subsystems  affirm  satisfaction  of 
training  needs  and  requirements. 

Provided  that  all  concerned  ultimately  "sign  off"  on  the  revised 
products,  there  will  be  across-the-board  satisfaction  of  everyone's 
''bottom  line."  Each  Subsystem  is  pleased,  because  it  has  met  all  of 
its  performance  requirements.  Command  is  pleased  not  only  because 
it  has  met  its  personal  performance  requirements,  but  even  more 
because  the  total  training  system  has  been  effective.  Sponsor  is 
happy  because  the  training  needs  have  been  met.  Each  has  its  own 
point  of  view,  its  own  definitions  and  standards  of  "happiness." 
That  is  to  say,  each  faces  its  own  taxa  of  performance  requirements. 
There  is  a  direct  parallel  between  this  three-tiered  training 
structure  and  a  professional  baseball  club.  After  a  game,  the  center 
fielder  (for  example)  might  say,  "I'm  happy:  when  they  threw  it,  I 
hit  it,  and  when  they  hit  it,  I  caught  it."  The  manager  might  say, 
"I'm  happy:  we  won."  The  owner  might  say,  "I'm  happy:  by 
winning,  and  by  continuing  to  win,  we've  brought  paying  customers 
into  the  ballpark." 

This  discussion,  and  Figure  24  on  which  it  is  based,  of  course  is  very 
much  oversimplified.  Evaluation  and  design  don't  (or  shouldn't)  actually 
proceed  in  three  distinct  waves.  Rather,  they  are  continual  or  even 
continuous  activities  that  always  are  blends  of  potentiality,  process,  and 
product  issues.  Also,  there  truly  is  nothing  "final"  about  a  system  design  or 
a  system  evaluation,  until  the  system  ultimately  goes  out  of  business.  All 
functioning  systems  always  admit  the  possibility  of  improvement,  and  so  always 
admit  the  need  for  additional  evaluation  and  additional  design.  System 
technology,  after  all,  steadily  improves.  As  it  does,  formerly  state-of-art 
system  designs  approach  obsolescence,  a  trend  that  can  only  be  halted  by 
modifying  the  system's  design  to  incorporate  the  newer  technology.  Evaluation 
and  design  are  forever  interlinked  and  forever  functioning  within  any 
operating  system.  Evaluation  and  design  are  the  two  threads  that  weave  the 
fabric  of  a  total  Systems  Analytic  Process. 
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V.  RECOMMENDATIONS  FOR  FURTHER  DEVELOPMENT 
AND  APPLICATIONS 


As  noted  in  Section  II,  the  ultimate  objective  of  this  APM  development 
work  is  to  provide  users  with  a  uniform,  thorough,  adaptive  and  efficient 
procedure  to  aid  in  the  process  of  deriving  the  most  meaningful  design 
specification  requirements,  design  specifications,  evaluation  measurement 
requirements  or  evaluation  measures  for  any  planned  or  existing 
human-machine  system  (particularly  a  training  system).  In  keeping  with  that 
objective,  the  APM  development  project  so  far  has  resulted  in  the  following 
main  products: 

°  An  annotated  bibliography  of  the  manned  systems  measurement 
literature  (244  documents  cited  and  abstracted). 

°  An  initial  analytic  process  model  for  systems  design  and 
measurement . 

0  A  generalizable  taxonomy  of  measurable  attributes  for  the 
surveillance  function  found  in  many  manned  systems. 

0  A  generalizable  model  of  training  systems  and  their  component 
subsystems. 

°  Generalizable  taxonomies  for  the  curriculum  development  (or  design) 
and  trainer  training  (enabling)  subsystems  of  training  systems. 

0  A  sample  application  of  the  model  for  deriving  effectiveness' 

measures  for  Learner  Testing  Specifications  within  the  emerging 
blFVTS ,  including  a  generalizable  taxonomy  and  application 
procedures . 

0  A  sample  application  of  the  model  for  establishing  Training  System 
design  requirements  for  the  Army's  new  9mm  Handgun,  including  a 
generalizable  taxonomy  and  application  procedures. 

°  An  untested  procedural  outline  of  a  computer-based  application  of 
the  APM  for  training  systems  measurement  (using  an  Apple  II 
computer)  . 

This  list  represents  considerable  progress  in  an  area  that  has  been  so 
difficult  to  standardize  for  so  many  years.  The  model  development  is  just  now 
beginning  to  yield  the  kinds  of  guidelines  and  sample  applications  that  soon 
will  allow  it  to  be  learned  and  applied  by  those  who  are  its  potential  users. 
To  help  bring  the  APM  development  process  to  the  point  of  application  by  the 
ultimate  users,  several  tasks  need  to  be  carried  out.  These  are  listed  in 
Figure  25.  It  is  noted  that  the  items  are  not  necessarily  independent  of  each 
other,  but  some  tasks  can  be  seen  as  being  a  part  of  others.  They  are  listed 


DEVELOPMENT 

1.  Further  Develop  the  APM  for  Training  Systems  Measurement 

2.  Further  Develop  the  APM  for  Training  Systems  Specification 

3.  Further  Develop  Training  Systems  Taxonomies  for  the  APM 

4.  Further  Develop  Procedures  for  Measures  Derivation 

5.  Further  Develop  Procedures  for  Identifying  the  Human  Operators' 

Contribution  to  Specifications  or  Measures 

6.  Develop  Procedures  for  Differentiating  Specifications  and 
Measures  in  Terms  of  Weighting  Factors,  Delay  Tolerance  and 
Criticality 

APPLICATION 

7.  Apply  the  APM  (Measurement)  to  an  Existing  Army  Training  System 

8.  Apply  the  APM  (Design  Specifications)  to  a  New  Army  Training 
System 

9.  Improve  Procedures  for  User  Application  of  the  APM  (Measurement  i> 
Design) 

10.  Test  Procedures  for  User  Application  of  the  APM  (Measurement  4 
Design) 

11.  Improve  and  Test  Routinised  Procedures  Using  Interactive  Computer 

12.  Develop  and  Test  a  Training  Program  for  APM  Users 

13.  Train  Users  to  Apply  the  APM  (Measurement  4  Design) 

14.  Provide  Mechanism  for  Future  Modification/Improvement  of  the  APM 


Figure  25.  Recommended  Further  Research  on  APM  for  Training 
Systems  Specification  and  Measurement 


separately,  however,  because  they  are  identifiably  distinct  or  limited  in  their 
purpose.  For  example,  Item  1  can  be  seen  as  a  general  task,  within  which  are 
several  others  including  part  of  Item  3  and  all  of  Item  4.  Item  4,  in  turn,  can 
be  seen  as  having  within  it  several  others  including  parts  of  Items  5  and  6. 
Likewise,  Item  2  can  be  seen  as  a  general  task,  within  which  one  could  also 
place  part  of  Items  3,  5  and  6. 

Item  1.  Further  Develop  the  APM  for  Training  Systems  Measurement 

The  purpose  of  this  research  item  is  to  continue  developing  the  APM 
for  measurement  toward  the  ultimate  goal  of  providing  a  practical  and  useful 
end  product.  The  research  task  ultimately  should  address  all  stages  of  the 
model  in  order  to  extend  its  conceptualization  and  utility  in  each  of  its 
aspects.  The  research  method  should  include  model  application  with  several 
existing  or  developing  training  systems  to  verify  or  modify  its  component 
parts,  its  adequacy,  and  its  practicality.  The  effort  should  also  produce 
improved  and  amplified  procedures  and  guidelines  for  using  the  model. 

Item  2.  Further  Develop  the  APM  for  Training  Systems  Specification 

The  purpose  of  this  research  item  is  analogous  to  that  of  Item  1,  but 
as  applied  to  the  APM  for  specification.  Sample  applications  should  involve  the 
specification  of  several  new  training  systems.  All  component  parts  of  the 
specification  model  should  be  addressed;  concepts  and  definitions  should  be 
clarified  and  application  guidelines  should  be  developed.  In  addition,  the 
utility  of  the  model  for  assessing  previously  generated  design  requirements 
should  be  evaluated. 


Item  3.  Further  Develop  Training  Systems  Taxonomies  for  the  APM 

Currently,  generalized  taxa  for  any  Design  Subsystem  haVe  been 
developed  and  are  being  applied  to  BIFVTS  issues  for  verification,  A  second 
generalized  taxonomy  for  any  Enabling  Subsystem  is  also  completed.  Taxa  for 
the  additional  generalized  training  subsystems  should  be  developed  and  their 
applicability  to  BIFVTS  issues  should  be  verified  through  some  real  world 
comparison  or  evaluation,  by  subject-matter  experts  (SMEs)  or  otherwise.  The 
same  procedures  should  be  applied  as  used  in  prior  taxa  development  and 
verification  of  the  taxa  should  be  carried  out.  Taxa  should  be  completed  for 
each  of  the  training  subsystems  of  the  BIFVTS. 

Item  4.  Further  Develop  Procedures  for  Measures  Derivation 

One  of  the  most  difficult-to-develop  segments  of  the  APM  for 
measurement  is  that  in  which  measurable  attributes  are  determined  and  actual 
measures  are  selected  (Ref.:  Figure  1,  blocks  6  and  7).  Some  progress  has 
been  made  during  the  past  year  in  beginning  to  identify  specific  procedures 
for  measures  derivation.  More  needs  to  be  done,  however,  especially  if 
computer-aiding  is  to  be  used  for  this  segment.  In  particular,  the  analytic 
steps  leading  to  measures  derivation  involve  a  determination  of  issues  that 
imply  a  need  to  measure  particular  attributes. 


Additional  work  is  also  needed  to  incorporate  the  MOE-MOP 
development  steps  into  the  APM,  identify  the  relationships  among  MOE-MOP  for 
systems-subsystems  levels,  and  identify  the  relationships  among  the  machine 
and  human  components  in  terms  of  contribution  to  MOE-MOP  development  or 
specification.  This  subtask  can  build  on  the  current  BIFVTS  work,  by 
continuing  more  sample  or  partial  applications  to  increasingly  complex  issues  of 
military  training,  both  to  verify  and  to  improve  the  procedural  guidelines. 
This  is  analogous  to  the  earlier  process  of  generating  taxonomization  guidelines 
in  this  research  effort.  The  procedural  guidelines  allow  for  the  derivation  of 
MOE-MOP  specific  to  given  measurement  purposes  and  provide  for  direct 
MOE-MOP  (type)  selection  given  the  defined  measurement  purpose.  Those 
guidelines,  when  better  understood,  will  define  the  algorithm  for  measures 
derivation  that  then  can  be  included  in  the  computer-auded  process. 

Item  5.  Further  Develop  Procedures  for  Identifying  the  Human 
(Operators*  Contribution  to  Specifications  or  Measures 

One  aspect  of  the  measures  derivation  process  (Ref.:  Item  4) 
requiring  particular  attention  is  the  determination  of  operator-centered 
measures.  This  task  should  identify  the  many  (sometimes  subtle)  roles  of 
human  operators  in  the  total  system  measurement  and  evaluation  process. 
Researchers  should  apply  the  appropriate  techniques  of  Human  Factors 
Engineering  to  define  the  roles  and  influence  of  operators  in  the  various 
system  attributes  selected  for  specification  or  measurement.  Some  researchers 
using  the  model  in  the  future  may  require  assistance  in  maintaining  the 
necessary  awareness  of  the  human's  role,  so  that  their  derived  specifications 
and  measures  will  give  due  consideration  to  operator-determined  aspects  of 
system  performance. 

Item  6.  Develop  Procedures  for  Differentiating  Specifications 
and  Measures  in  Terms  of  Weighting  Factors,  belay 
Tolerance  and  Criticality  2 

The  practical  aspects  of  model  utilization  suggest  that  there  will  be 
times  when  the  full  range  of  specifications  or  measures  cannot  be  applied  in  its 
entirety,  and  that  choices  will  have  to  be  made  to  select  only  the  most 
important  ones  for  application.  To  aid  the  user  in  selecting  those 
specifications  or  measures  which  are  most  important,  a  procedure  should  be 
developed  which  helps  to  rate  or  rank  measures  in  accordance  with  such 
criteria  as:  criticality  to  mission  performance,  amenability  to  alternatives  or 
corrective  actions,  tolerance  to  postponement  or  delay  in  application,  or  other 
criteria  to  be  determined. 


It  em  7.  Apply  the  APM  for  Measurement  to  an  Existing  Army  System 

This  item  involves  extending  the  current  developments  of  the 
Curriculum  Development  Subsystem  of  the  BIFVTS  to  a  practical  application 
that  also  can  serve  as  a  validation  or  verification  exercise.  For  example,  the 
model  can  be  applied  in  the  context  of  the  Training  Effectiveness  Analysis 
(TEA)  of  the  BIFVTS  planned  by  USAIS/USAIC.  We  understand  that  the  TEA 


is  to  begin  in  4th  Qtr  FY  1982.  Current  developments  can  be  extended  to  the 
point  where  derivation  of  specific  measure  sets  for  the  effectiveness  of  the 
BIFVTS  subsystem  can  be  generated.  These  measure  sets  can  then  be 
examined  for  feasibility,  validity  and  utility  in  the  context  of  the  TEA.  It 
would  be  necessary  to  review  the  measurement  issues  to  be  addressed  in  the 
TEA  that  are  relevant  to  the  BIFVTS  curriculum,  and  one  or  more  of  those 
issues  should  be  selected  as  the  basis  for  this  task.  The  performance  taxa 
pertaining  to  each  selected  issue  and  the  measurement  derivation  guidelines 
should  be  used  to  generate  a  measures  hierarchy  for  each  selected  issue. 
Differences  between  APM  and  TEA  measures  should  be  examined  so  that 
reasons  can  be  determined  for  those  differences.  Commonalities  should  also  be 
noted  and  their  reasons  understood.  For  practical  reasons,  this  in-depth 
application  and  comparison  should  focus  on  a  narrow  aspect  of  the  BIFVTS, 
such  as  Basic  Gunnery  Learner  Testing.  This  task  would  also  provide  an 
opportunity  to  focus  on  human  operator  contributions  to  performance,  such  as 
motivation,  attitudes,  capabilities,  and  procedures  under  given  environmental 
administrative  and  other  constraints. 

Item  8.  Apply  the  APM  for  Design  Specification  to  a 

New  Army  Training  System 

Initial  extension  of  the  APM  to  the  specification  of  training  system 
requirements  was  begun  during  the  current  contract  year.  This  item  is 
designed  to  extend  and  further  explore  this  use  of  the  APM  as  a  tool  through 
application  to  another  new  (or  developing)  training  system.  The  purpose  is  to 
develop  further  and  to  verify,  to  the  extent  possible,  the  capability  of  the 
APM  process  to  serve  this  function.  This  can  be  done  through  the  "blind" 
development  of  training  system  specifications  for  a  new  or  developing  system  to 
be  selected  by  ARI.  A  comparative  assessment  can  be  carried  out  between 
design  specifications  or  specification  requirements  as  developed  by  using  the 
APM,  and  similar  specifications  or  requirements  as  developed  by  other 
methods,  such  as  the  Early  Training  Estimation  System  under  development  by 
the  ARI  Fort  Bliss  Field  Unit.  This  task  could  also  make  use  of  the  APM's 
capabilities  both  to  generate  new  design  specifications  and  to  aid  in  the 
diagnostic  assessment  of  previously  generated  design  specifications.  For  the 
same  practical  reasons  as  mentioned  under  Item  7,  this  comparative  application 
should  take  place  in  depth  over  a  narrow  aspect  of  the  new  training  system. 

Item  9.  Improve  the  Procedures  for  User  Application  of  the 

APM  (Measurement  and  Design  Versions) 


This  task  would  enlarge  and  improve  the  documented  procedures  and 
guidelines  to  be  followed  by  users  in  applying  the  APM.  It  would  also  help  to 
define  the  steps  and  algorithms  essential  to  a  computer-aided  application  of  the 
model.  The  effort,  in  general,  would  make  use  of  all  the  prior  and  concurrent 
applications  of  the  APM,  documenting  the  steps,  the  decision  rules,  the 
criteria  and  other  factors  that  are  essential  to  application  procedures.  One  set 
of  procedures  would  be  expected  for  the  development  of  performance 
taxonomies  (Figure  1,  blocks  1-3),  but  later  blocks  in  the  model  would  require 


This  task  is  preliminary  to  training  actual  users  in  the  application  of 
the  APM.  It  would  consist  of  providing  a  skilled,  but  APM-naive  analyst 
(such  as  a  "non-project"  member  of  the  Dunlap  staff)  with  the  procedures  and 
guidelines,  to  be  used  in  developing  effectiveness  measures  in  one  application 
and  design  specifications  in  a  second  application.  The  purpose  of  this  trial 
application  effort  is  to  refine  and  tlarify  the  procedures  prior  to  making  them 
available  to  users  through  computer-aided  facilities  and  training  programs 
(Ref.:  Items  11-13).  If  this  task  can  be  conducted  with  the  interactive 
computer  capability,  then  Item  11  becomes  unnecessary  as  it  would  only 
duplicate  this  one.  If  the  computer-aided  capability  is  unavailable,  then  this 
task  would  be  conducted  manually  with  printed  guidelines  and  instructions. 
The  final  product  of  this  task  should  be  an  updated  set  of  guidelines, 
procedures,  sample  applications,  taxonomies,  checklists,  algorithms  or  any 
other  available,  documented  aid  to  APM  application. 

Item  11.  Improve  and  Test  Routinized  Procedures 
~  Using  the  Interactive  Computer 

This  task  is  intended  to  build  upon  the  existing  computer-aided 
capability,  by  adding  flexibility,  instruction  options,  thoroughness,  and  data 
bases  to  the  rudimentary  computer-based  model.  It  would  also  include  a  test 
of  the  procedures  by  a  skilled,  but  APM-naive  analyst  (as  in  Item  10),  to 
determine  where  modifications  are  required  to  clarify,  simplify  or  otherwise 
improve  their  value  and  ease  of  use. 

Item  12.  Develop  and  Test  a  Training  Program  for 
APM  Users 


This  task  should  include  whatever  remaining  development  effort  is 
necessary  to  yield  routinized  (computer-aided)  procedures  for  application  by 
users.  It  should  also  include  a  test  of  those  procedures  by  user  personnel. 
Among  others,  likely  candidate  "users"  for  this  test  would  be  administrators 
and  training  developers  of  the  USAIS.  This  task  should  consider  the 
development  of  training/usage  handbooks  or  guidelines  for  users;  contractor 
development  and  conduct  of  user-directed  workshops;  and  other  techniques  for 
user  training  in  the  APM  process  and  its  application.  The  task  should  also 
consider  the  training  /usage  of  the  APM  process  by  various  personnel  types 
and  for  both  possible  purposes  (evaluation  of  training  systems  and 
specification  of  training  systems).  The  eventual  procedures  for  training  of 
users  and  for  their  usage  of  the  APM  should  be  as  programmatic, 
proceduralized,  routinized  and  simplified  as  possible.  Advantage  should  be 
taken  of  the  model's  already  demonstrated  capabilities  for  flow  charting, 
programming,  indexing  and  other  operations  which  make  it  readily  reducible  to 
routinized  procedures.  The  user  field  test  of  these  procedures  should  aim  to 


confirm  their  completeness  and  accuracy,  their  utility  (based  on  the  acceptance 
by  users)  and  their  validity  (based  on  the  quality  and  comprehensiveness  of 
products) . 


Item  13.  Train  Users  to  Apply  the  APM  (Measurement 

and  Design 

This  task  is  one  in  which  the  previously  developed  and  validated 
training  program  (Ref.:  Item  12)  is  placed  into  operation  to  promulgate  the 
APM  as  an  analytic  tool  among  potential  users.  It  requires  the  sponsorship  of 
a  promulgating  agency,  which  will  be  identified  for  this  purpose  by  ARI.  One 
possible  agency  to  be  considered  for  this  dissemination  function  is  the 
U.S.  Army  Training  and  Doctrine  Command  (TRADOC).  Administrative 
preparations  for  this  task  item,  in  the  form  of  preparatory  explorations  and 
inlormation  exchange  with  the  possible  promulgating  agencies,  should  begin  in 
the  near  future,  even  though  the  task  itself  is  not  likely  to  begin  until  a  later 
date. 


The  task  should  identify  specific  users  in  specific  agencies,  and 
plans  for  administering  the  APM  training  program  should  be  developed.  A  key 
office  should  be  designated  to  administer  and  coordinate  the  subsequent 
training  activities,  and  should  be  accessible  for  future  inquiries  from  trainees 
and  users.  A  cadre  of  trained  instructors  and  an  instructor  training  program 
should  also  be  established  as  a  continuing  entity.  One  of  the  tasks  of  the 
APM  program  administrator  should  also  be  to  serve  as  an  information  exchange 
and  documentation  center  for  users.  The  mechanisms  and  procedures  for  those 
administrative  and  technical  capabilities  should  be  designed  as  part  of  this  task 
item. 

Item  14.  Provide  Mechanisms  for  Future  Modification  or 

Improvement  of  the  APH 

To  aid  the  promulgating  office  in  carrying  out  its  functions  as  an 
information  exchange  and  documentation  center  for  users,  a  technical  capability 
should  be  established  to  include  analysts,  programmers  and  the  interactive 
computer  facility  for  APM  applications.  These  resources  would  be  used  to 
maintain  the  APM,  its  associated  data  bases,  and  its  various  application 
routines  and  records.  Users  can  then  be  helped  with  the  latest  fund  of 
knowledge,  techniques  and  experience.  A  current  mailing  list  of  users  would 
allow  for  rapid  distribution  of  application  documents,  program  modifications, 
information  requests,  and  suggestions  to  improve  the  value  of  this  resource  to 
users . 


One  of  the  most  important  sources  of  ideas  to  keep  the  APM  as  a 
useful  and  valuable  resource  is  the  user  population  itself.  A  method  should  be 
established  for  permitting  and,  in  fact,  encouraging  users  to  supply  feedback 
of  application  ideas  to  the  promulgating  office,  with  the  assurance  that  those 
ideas  will  be  given  careful  consideration  and  dissemination  in  the  most 
appropriate  manner.  Dissemination  can  employ  a  newsletter,  a  "new 
application"  description,  a  revised  data  base  or  even  a  message  for  CRT 
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display  when  users  log  on  to  use  the  model.  In  fact,  one  way  in  which  users 
can  conveniently  communicate  briefly  with  the  APM  central  office  could  be  via 
the  interactive  terminal,  such  as  by  typing  in  messages  or  inquiries  after 
using  the  program  or  by  pausing  during  use  when  a  problem  is  encountered. 
The  specific  mechanisms  for  keeping  the  APM  current  and  useful  should  be 
determined  and  implemented  in  this  task  item. 
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APPENDIX  A. 


Examination  of  Design  Subsystem  Performance 
Potentialities  Taxa  to  Assess  Their  Relevance  to 
Evaluation  of  Learner  Testing  Specifications 


"Identifying  Goals  and  Priorities  for  Training"  (Taxon  1.1) 

1.1. 1.1  "Identifying  types  of  achievements  relevant  to  the  intended 
job . " 


The  job-relevant  achievements  ultimately  represent  the 
subject  matter  of  learner  testing.  The  single  most  important 
reason  for  testing  learners  is  to  determine  whether  they  are 
ready  to  do  the  job  for  which  they  have  been  trained.  If  the 
Design  subsystem  does  not  have  the  capability  of  identifying 
accurately  the  job-relevant  achievements,  the  testing 
specifications  almost  certainly  will  be  deficient.  If  important 
achievements  are  overlooked  in  the  training  design,  they  will 
also  not  be  covered  by  any  specified  test.  If  non-relevant 
achievements  are  included  in  the  training  design,  then 
corresponding  non-relevant  tests  will  be  included  in  the 
specification.  In  either  case  the  tests  specified  would  produce 
distorted  images  of  learners'  qualifications  for  the  job. 

Conclusion:  This  taxon  is  relevant  to  the  measurement 

application. 

Note  that  this  conclusion  automatically  insures  the 
relevance  of  taxon  1.1.1  ("Defining  the  total  scope  of  learning") 
and  taxon  1.1  ("Identifying  goals  and  priorities"). 

1.1. 1.2  "Analyzing  achievements  to  determine  suitability  for  training." 

The  testing  specifications  must  include  provisions  for 
assessing  the  candidate  job  performer's  abilities  relative  to 
every  achievement  deemed  relevant  to  the  job.  This  is  true  of 
all  achievements  that  are  selected  as  suitable  for  training.  It  is 
equally  true  of  those  achievements  deemed  not  suitable  for 
training:  those  become  cast  as  prerequisites  for  the  job.  It  is 
just  as  necessary  to  know  whether  a  prospective  learner 
possesses  the  prerequisites  as  it  is  to  know  whether  the  learner 
has  achieved  the  abilities  addressed  in  training.  The  Design 
subsystem's  capabilities  for  assessing  whether  or  not  a 
job-relevant  achievement  is  suitable  for  training  will  have  a 
major  impact  on  the  subsystem's  performance  in  specifying 
learning  content.  It  should  not  affect  the  system's  performance 
in  specifying  testing.  Of  course,  if  the  content  is  poorly 
analyzed,  the  tests  probably  will  disclose  that  the  training  fails 
to  qualify  learners  as  job  performers. 

Conclusion:  This  taxon  is  not  relevant  to  the  application. 
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1. 1.2.1 


"Determining  necessary  levels  of  achievement. 


Learner  testing  is  supposed  to  disclose  not  only  whether 
the  candidate  job  performer  can  carry  out  the  actions  relevant 
to  the  job,  but  also  whether  he  or  she  can  do  so  well  enough  to 
meet  the  job's  requirements.  For  example,  some  job  mignt 
require  a  performer  to  lift  a  50-lb.  weight  and  hold  it  at 
shoulder  height  for  a  period  of  time.  It  would  be  critical  for 
training  design,  personnel  selection,  and  learner  testing  to 
determine  whether  that  "period  of  time"  is  instantaneous,  a  few 
seconds,  several  minutes,  or  whatever.  If  the  job's 
requirements  called  for  holding  the  weight  for,  say, 

15  seconds,  it  would  be  inappropriate  to  test  the  learner  by 
requiring  that  he  or  she  hold  it  for  only  5  seconds;  it  would  be 
equally  wrong  to  conduct  a  test  that  required  holding  the 
weight  for  a  full  minute.  The  former  test  would  "qualify"  some 
learners  who  actually  do  not  meet  job  requirements;  the  latter 
test  would  "disqualify"  some  who  can  really  do  the  job.  Thus, 
a  Design  subsystem  that  is  not  capable  of  accurately 
determining  necessary  levels  of  achievement  very  likely  will 
produce  test  specifications  that  are  deficient. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

Note  that  this  insures  as  well  the  relevance  of  taxon  1.1.2 
("Stating  the  ultimate  intended  outcomes  of  learning"). 

1.1. 2. 2  "Determining  existing  levels  of  achievement." 

This  capability  explicitly  calls  for  testing.  It  requires 
that  the  current  status  of  the  candidate  learner  populations' 
abilities  relevant  to  the  job  be  determined.  This  implies  that  at 
least  some  testing,  even  if  only  informal,  be  applied  to  at  least 
some  representatives  of  those  populations. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

1.1. 2. 3  "Discerning  discrepancies  between  existing  and  necessary  levels 
of  achievement." 

This  requirement  pertains  to  the  Design  subsystem's  ability 
to  accurately  assess  differences  between  what  the  populations  of 
candidate  job  performers  can  do  now,  and  what  they  need  to  be 
able  to  do  to  perform  the  job  satisfactorily.  That  ability 
demands  access  to  data  that  can  be  acquired  through  testing, 
and  demands  the  subsystem's  ability  to  apply  those  data 
correctly.  But,  it  has  no  direct  bearing  on  the  test 
specifications  themselves. 

Conclusion:  This  taxon  is  not  relevant  to  the  application. 
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1. 1.3.1  "Assessing  goal  importance." 


The  absolute  or  relative  importance  of  the  various  training 
goals  have  no  impact  on  the  need  to  test  all  achievements 
relevant  to  the  job. 

Conclusion:  This  taxon  is  not  relevant  to  the  application. 

1.1. 3. 2  "Insuring  availability  of  a  relative  (numeric)  goal  ranking 
scheme. " 

The  comment  given  under  taxon  1.1. 3.1  also  applies  here. 

Conclusion:  This  taxon  is  not  relevant  to  the  application. 

1.1. 4.1  "Insuring  availability  of  a  hierarchy /taxonomy  of  goals." 

A  Design  subsystem  must  be  able  to  classify  its  training 
goals  and  objectives  in  a  way  that  permits  the  kinds  of  actions 
required  of  learners  and  job  performers  to  be  identified 
accurately.  This  is  of  particular  significance  to  test 
specification,  for  without  a  suitable  classification  scheme,  the 
wrong  type  of  test  might  be  specified.  For  example,  if  an 
objective  requires  a  learner  to  demonstrate  some  physical  act,  it 
would  not  be  sufficient  to  test  the  learner's  achievement  of  that 
objective  only  by  requiring  him  or  her  to  state  the  physical 
activity.  That  test  would  disclose  whether  the  learner  knows 
what  is  needed,  but  it  wouldn't  disclose  whether  the  learner 
can  really  do  what  is  needed. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

Note  that  taxon  1.1.4  ("Establishing  a  basis  for  specifying 
performance  objectives")  is  thus  also  relevant. 

1.1. 4. 2  "Stating  goals  in  terms  that  imply  behaviors." 

If  a  Design  subsystem  is  unable  to  state  its  training  goals 
in  behaviorally  relevant  terms,  it  is  unlikely  that  the  actions 
required  of  learners  will  be  identified  correctly  or  expressed  in 
a  fashion  insuring  they  will  be  observable  and  quantifiable.  If 
the  required  actions  are  unclear,  then  the  actions  to  be  tested 
also  will  be  unclear.  Proper  test  specification  demands 
behaviorally  defined  goals  and  objectives. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

"Establishing  Performance  Objectives"  (Taxon  1.2) 

1.2. 1.1  "Defining  the  performance  action." 

If  action  cannot  be  precisely  defined,  then  the  learner's 
ability  to  produce  the  action  cannot  be  exercised  in  a  test. 
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Conclusion:  This  taxon  is  relevant  to  the  application. 


Note  that  taxa  1.2.1  ("Stating  what  learners  will  be  able  to 
do  or  how  they  will  be  expected  to  behave  after  completing 
training")  and  1.2  ("Establishing  performance  objectives") 
automatically  are  relevant  as  well. 

1.2. 1.2  "Defining  the  performance  conditions." 

If  the  performance  conditions  required  for  performing  the 
action  on  the  job  cannot  be  precisely  defined,  then  no 
assurance  can  be  had  that  the  specified  tests  will  exercise  the 
learner's  ability  under  the  appropriate  set  of  circumstances. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

1.2. 1.3  "Defining  the  performance  standard." 

If  the  standard  for  "good  enough"  performance  cannot  be 
precisely  defined,  the  test  specifications  likely  will  not  provide 
for  accurate  interpretation  of  the  adequacy  of  the  learner's 
action. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

1.2.2. 1  "Developing  pre-test  items." 

Pre-test  items  form  an  integral  portion  of  the  test 
specifications. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

Thus,  taxon  1.2.2  ("Providing  a  basis  for  objectively 
assessing  learner  performance")  also  is  relevant. 

1.2. 2. 2  "Developing  post-test  items." 

Post-test  items  form  an  integral  portion  of  the  test 
specifications. 

Conclusion:  This  taxon  is  relevant  to  the  application, 
c.  "Analyzing  Performance  Objectives"  (Taxon  1.3) 


1.3. 1.1  "Revealing  present  conditions  of  learning." 

1.3. 1.2  "Identifying  essential  prerequisites." 

1.3. 1.3  "Identifying  supportive  prerequisites." 

Test  specifications  must  provide  for  diagnosing  the  cause 
of  any  deficiency  in  a  learner's  performance.  Such  deficiencies 
might  arise  because  a  learner  fails  to  accomplish  some 

?irerequisite  to  a  given  objective,  i’or  example,  the  learner  may 
ail  to  recall  something  that  is  relevant  to  learning  (a  deficiency 
in  the  present  conditions  of  learning);  or  the  learner  may  not 
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have  mastered  some  achievement  that  is  an  essential  milestone 
toward  the  objective  (a  deficiency  in  an  essential  prerequisite); 
or  the  learner  may  not  have  mastered  some  achievement  that, 
while  not  essential  for  the  objective,  would  have  facilitated 
attainment  of  the  objective  (a  deficiency  in  a  supportive 
prerequisite).  Unless  these  types  of  prerequisites  can  be 
identified,  they  won't  be  reflected  in  the  test  specifications. 

Conclusion:  These  three  taxa  are  all  relevant  to  the 

application.  ~  ~  — 

Note,  therefore,  that  taxon  1.3.1  ("Identifying 
prerequisites  to  learning")  and  taxon  1.3  ("Analyzing 
performance  objectives")  also  are  relevant. 

1.3. 2.1  "Developing  entry  test  items." 

Entry  test  items  form  an  integral  portion  of  the  test 
specifications . 

Conclusion:  This  taxon  is  relevant  to  the  application. 

Taxon  1.3.2  ("Providing  a  basis  for  objectively  assessing  a 
candidate  learner's  suitability)  thus  also  is  relevant  to  the 
application. 

1.3.3. 1  "Discerning  independence  between  objectives." 

1.3. 3. 2  "Discerning  dependency  between  objectives." 

1.3. 3. 3  "Discerning  a  supportive  relationship  between  objectives." 

These  potentialities  express  the  capability  that  the  Design 
subsystem  must  have  for  determining  the  sequence  in  which 
performance  objectives  should  be  achieved  during  training. 
Certain  objectives  logically  must  be  achieved  before  others  are 
pursued.  In  other  cases,  the  achievement  of  a  given  objective 
will  aid  a  learner's  pursuit  of  another  objective  even  though  it 
is  not  essential  for  that  pursuit.  The  Design  subsystem  must 
have  the  capability  of  identifying  these  dependency 
relationships  if  it  is  to  structure  the  training  efficiently  and 
effectively.  However,  the  dependency  among  objectives  really 
has  no  direct  bearing  on  the  testing  of  those  objectives. 

Conclusion:  These  three  taxa  are  not  relevant  to  the 

application. 

1.3. 4.1  "Identifying  information  type  objectives." 

1.3. 4. 2  "Identifying  mental  skills  type  objectives." 

1.3. 4. 3  "Identifying  physical  skills  type  objectives." 

1.3. 4. 4  "Identifying  attitude  type  objectives." 

The  comment  given  above  in  reference  to  taxon  1.1. 4.1  is 
applicable  here  as  well.  Unless  the  Design  subsystem  can 
accurately  classify  its  performance  objectives  in  terms  of  the 
type  of  learning  called  for,  the  test  specifications  may  not 
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insure  that  the  proper  type  of  testing  takes  place. 
Measurement  of  a  learner's  achievement  of  a  mental  skills 
objective,  for  instance,  requires  a  mental  skills  test; 
achievement  of  a  physical  skills  objective  must  be  measured  by 
a  physical  skills  text;  and  so  forth. 

Conclusion:  These  four  taxa  are  relevant  to  the 

application .  ~  “ -  — 

Note  that  taxon  1.3.4  ("Classifying  objectives  in  terms  of 
the  domains  of  learning  involved")  automatically  is  relevant  as 
well. 

1.3.5. 1  "Identifying  overt  steps." 

1.3. 5.2  "Identifying  covert  steps." 

1.3. 5. 3  "Identifying  unconscious  steps." 

The  Design  subsystem  must  have  the  capability  of 
analyzing  the  actions  required  of  the  performance  objectives  to 
identify  the  constituent  steps  that  make  up  the  action.  This  is 
essential  for  structuring  the  training  content  in  a  way  that 
fully  explains  each  task  to  the  learners,  and  that  enables  the 
learner  to  practice  performing  the  task.  The  capability  is  also 
essential  for  insuring  proper  testing  of  learners,  since  the 
testing  should  diagnose  performance  deficiencies.  In  other 
words,  if  a  learner  fails  in  performing  a  task  on  which  he  or 
she  is  being  tested,  it  should  be  possible  to  determine  which  of 
the  overt,  covert,  or  unconscious  steps  led  to  the  failure. 
This  is  a  significant  challenge  to  the  test  designer,  since  he  or 
she  must  devise  ways  to  translate  the  covert  and  unconscious 
steps  into  observable,  quantifiable  behaviors.  This  often 
requires  that  the  learner-being-tested  state  aloud  the  results  of 
mental  processes,  decisions,  recognitions,  etc.,  that  proceed  in 
silence  under  normal  on-the-job  conditions.  But  certainly,  the 
test  designer  cannot  develop  means  of  diagnosing  the  learner's 
performance  of  the  constituent  steps  unless  those  steps  can  be 
identified. 

Conclusion:  These  three  taxa  are  relevant  to  the 

application. 

Note  that  taxon  1,3.5  ("Identifying  component  steps  or 
processes  within  the  objectives")  automatically  is  relevant  as 
well. 


"Defining  Training  Content"  (Taxon  1.4) 

1.4. 1.1  "Defining  propositions,  names,  facts,  etc.,  that  are  relevant  to 
information  objectives." 

This  capability  is  essential  to  insure  that  the  Design 
subsystem  can  accurately  select  the  content  to  be  provided  to 
learners  to  assist  them  in  achieving  information-type  objectives. 
This  is  an  extremely  critical  capability  bearing  on  how  well  the 


subsystem  specifies  the  learner  presentation/demonstration 
activities.  However,  how  the  learner  learns  is  of  no  real 
significance  to  the  specification  of  means  for  determining 
whether  the  learner  has  learned.  Indeed,  test  designers  often 
err  by  producing  test  specifications  that  measure  how  well  the 
learner  recalls  his  or  her  learning  experiences  and  the  material 
to  which  he  or  she  was  exposed  rather  than  measuring  how  well 
the  learner  has  achieved  the  stated  objectives.  If  the 
instructional  content  provided  to  support  a  given 
information-type  objective  was  poorly  chosen,  the  tests  specified 
for  that  objective  probably  will  disclose  widespread  performance 
deficiencies  among  learners.  But  the  central  fact  is  that  the 
objectives-referenced  tests  should  be  independent  of  the 
instructional  content. 

Conclusion:  This  taxon  is  not  relevant  to  the  application. 

1.4. 1.2  "Defining  concepts,  rules,  algorithms,  etc.,  relevant  to  mental 
skills  objectives." 

1.4. 1.3  "Defining  movements,  timings,  actions,  etc.,  relevant  to 
physical  skills  objectives." 

1.4. 1.4  "Defining  values,  choices,  etc.,  relevant  to  attitude  objectives." 

The  comments  given  above  in  reference  to  taxon  1.4. 1.1 
apply  here  as  well. 

Conclusion:  These  three  taxa  are  not  relevant  to  the 

application. 

"Assessing  applicability  of  Job  Performance  Aids." 

"Assessing  applicability  of  Self-Teaching  Exportable  Packages." 
"Assessing  applicability  of  Formal  On-the-job  Training." 
"Assessing  applicability  of  Installation  Support  School.” 
"Assessing  applicability  of  Resident  School." 

The  Design  subsystem  must  possess  these  five  capabilities 
if  it  is  to  do  a  proper  job  in  choosing  the  settings  for  the 
various  learning  activities.  However,  the  issue  of  where 
training  takes  place  has  no  bearing  on  the  specification  of  tests 
to  determine  whether  the  training  was  successful. 

Conclusion:  These  five  taxa  are  not  relevant  to  the 

application. 

1.4.3. 1  "Sequencing  in  dependency  order." 

1.4. 3. 2  "Sequencing  in  supportive  relationship  (efficiency)  order." 

The  Design  subsystem  must  possess  these  capabilities  to 
insure  that  the  performance  objectives  will  be  pursued  in  a 
logical,  efficient  sequence.  That  is  absolutely  essential  if  the 
learning  experiences  are  to  be  as  effective  as  possible.  But 
the  order  in  which  objectives  are  pursued  should  have  no 
bearing  on  the  tests  to  be  used  to  determine  whether  the 
objectives  are  met. 


1.4. 2.1 

1.4. 2. 2 

1.4. 2. 3 

1.4. 2. 4 

1.4. 2. 5 
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Conclusion:  These  two  taxa  are  not  relevant  to  the 

application.  ~~ 


e.  "Defining  Training  Procedures"  (Taxon  1.5) 


1.5. 1.1 

1.5. 1.2 

1.5. 1.3 

1.5. 1.4 

1.5. 1.5 


"Defining  facilitator  preparation  activities." 

"Defining  facilities  preparation  requirements." 

"Defining  attention  gaining  procedures." 

"Defining  objective-informing  procedures." 

"Defining  procedures  for  stimulating  learners'  recall  of 
prerequisites. " 


All  of  these  capabilities  must  be  present  in  the  Design 
subsystem  if  it  is  to  be  able  to  specify  appropriate  and  efficient 
means  of  assisting  learners  to  prepare  for  learning.  However, 
none  of  these  capabilities  bear  on  the  subsystem's  work  in 
specifying  appropriate  means  of  testing  learners. 


Conclusion:  These  five  taxa  are  not  relevant  to  the 

application. 

1.5. 2.1  "Defining  presentation  media/media  alternatives." 

1.5. 2. 2  "Defining  procedures  and  requirements  for  placing  emphasis." 

1.5. 2. 3  "Defining  procedures  for  enhancing  learners'  comprehension  and 
retention. " 


All  of  these  capabilities  must  be  present  in  the  Design 
subsystem  if  it  is  to  be  able  to  specify  appropriate  and  efficient 
means  of  presenting/demonstrating  the  training  content  and 
material  to  the  learners.  However,  none  of  these  capabilities 
bears  on  the  subsystem's  work  in  specifying  appropriate  means 
of  testing  learners. 

Conclusion:  These  three  taxa  are  not  relevant  to  the 

application. 

1.5.3. 1  "Defining  procedures  for  enhancing  learners'  participation/ 
involvement. " 

1.5. 3. 2  "Defining  procedures  for  eliciting  the  performance  from 
learners. " 

1.5. 3. 3  "Defining  procedures  for  providing  learning  guidance  to 
learners. " 


These  three  capabilities  must  be  present  in  the  Design 
subsystem  if  it  is  to  be  able  to  specify  appropriate  and  efficient 
means  of  assisting  the  learners  to  practice/apply  what  they  are 
learning.  However,  none  of  these  capabilities  bears  on  the 
subsystem's  work  in  specifying  appropriate  means  of  testing 
learners. 

Conclusion:  These  three  taxa  are  not  relevant  to  the 

application . 
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"Defining  procedures  for  providing  learning  feedback  to 
learners. M 

"Defining  procedures  for  assessing  sufficiency  of  practice." 

The  two  capabilities  also  are  essential  to  insure  that  the 
Design  subsystem  will  be  able  to  specify  appropriate  and 
efficient  practice  for  the  learners.  The  key  ingredient  that 
these  capabilities  supply  is  learner  feedback,  i.e.,  the  ability 
to  inform  the  learner  of  how  he  or  she  is  doing  in  practice,  and 
to  help  the  learner  and  facilitator  judge  whether  the  learner  has 
practiced  enough,  at  least  for  the  time  being.  These  abilities 
demand  some  form  of  testing.  Usually,  this  is  learner 
self-testing,  perhaps  under  observation  and  "scoring"  by  a 
facilitator. 

Conclusion:  These  two  taxa  are  relevant  to  the 

application.  ™—  ”~ 

Note  that  this  automatically  implies  that  taxon  1.5.3 
("Specifying  application  procedures")  and  taxon  1.5  ("Defining 
training  procedures")  also  are  relevant. 

"Defining  procedures  to  assess  the  proper  domain  of  learning." 

This  capability  aims  at  one  key  ingredient  of  learner 
testing,  namely,  the  ability  to  insure  that  the  proper  kinds  of 
tests  are  used  for  a  given  class  of  learning  objectives. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

Note  that  taxon  1.5.4  ("Specifying  learner  evaluation 
procedures")  therefore  is  relevant  as  well. 

"Defining  procedures  to  assess  the  appropriate  action/ 
behavior. " 

This  capability  goes  to  the  heart  of  another  key  ingredient 
of  learner  testing,  namely,  the  ability  to  insure  that  the  tests 
require  the  learner  to  demonstrate  the  actions  that  are 
appropriate  for  the  performance  objectives. 

Conclusion:  This  taxon  is  relevant  to  the  application. 
"Defining  procedures  to  insure  assessment  reliability." 

This  capability  focuses  on  yet  another  key  ingredient  of 
learner  testing,  namely,  the  ability  to  insure  that  the  tests  are 
conducted  subject  to  conditions  and  performance  standards  that 
reflect  real  job  requirements. 


Conclusion:  This  taxon  is  relevant  to  the  application. 


valuating  the  Curriculum"  (Taxon  1.6) 

6.1.1  "Entry-level  testing  of  representative  candidate  learners." 


In  order  to  validate  the  assumptions  underlying  the 
curriculum,  the  Design  subsystem  must  be  able  to  ascertain 
whether  the  candidate  learner  populations  actually  satisfy  the 
prerequisites  specified  for  the  training.  Usually,  if  not  always, 
this  entails  some  type  of  testing  of  at  least  some  representative 
members  of  those  populations.  The  ability  to  construct  and 
apply  such  tests  is  an  ingredient  of  the  subsystem's  overall 
performance  in  specifying  learner  testing. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

Note  that  this  implies  that  taxa  1.6.1  ("Validating /revising 
the  bases  for  the  developing  curriculum")  and  1.6  ("Evaluating 
the  curriculum")  also  are  relevant. 

6.1.2  "Post-training  testing  of  representative  job  incumbents." 

The  training  is  supposed  to  prepare  the  learners  to  do  the 
job  for  which  the  training  was  designed.  Post-training  tests 
specified  by  the  Design  subsystem  are  supposed  to  provide  a 
basis  for  determining  whether  the  learners  are  in  fact  prepared 
to  do  the  job.  The  post-tests,  in  other  words,  are  supposed  to 
predict  job  performance.  Whenever  possible,  it  is  desirable  to 
validate  the  post-tests  by  applying  them  to  current 
(satisfactory)  job  incumbents.  The  ability  to  do  so  clearly  is 
an  ingredient  of  the  subsystem's  overall  performance  in 
specifying  learner  testing. 

Conclusion:  This  taxon  is  relevant  to  the  application. 

6.2.1  "Obtaining  error  data." 

One  key  question  to  be  faced  at  the  final  stages  of  a 
curriculum  development  effort  is:  "Given  a  group  of  learners 
who  have  received  training  in  accordance  with  the  curriculum, 
how  well  do  those  learners  perform?"  Part  of  the  answer  is 
found  by  administering  specified  post-tests  to  a  sample  of  (pilot 
test)  learners.  Such  testing  produces  error  data,  disclosing 
which  objectives  or  portions  of  objectives  were  not  achieved  by 
significant  numbers  of  learners.  The  ability  to  obtain  this 
error  data  is  an  ingredient  of  the  subsystem's  overall 
performance  in  specifying  learner  testing. 

Conclusion;  This  taxon  is  relevant  to  the  application. 

Note  that  taxon  1.6.2  ("Validating /revising  the  developing 
curriculum  itself")  thus  also  is  relevant. 


1.6. 2. 2  "Obtaining  design  feedback." 
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Design  feedback  consists  of  general  and  specific  criticisms 
of  the  curriculum  by  representative  "users"  of  the  curriculum, 
i.e.,  learners  and  facilitators.  Like  error  data*  design 
feedback  typically  is  obtained  in  pilot-test  settings,  often  at 
virtually  the  same  time  that  learner  post-testing  occurs. 
However,  the  solicitation  of  design  feedback  is  not  a  test,  nor 
is  it  in  any  way  part  of  the  curriculum's  test  specifications. 

Conclusion:  This  taxon  is  not  relevant  to  the  application. 

1.6. 2. 3  "Correlating  post-test  data  with  subsequent  job  performance." 

The  ultimate  evaluation  of  a  new  curriculum  is  found  in  the 
learners'  job  performance  subsequent  to  training.  The  Design 
subsystem  must  have  the  capability  of  obtaining  measures  on  job 
performance,  at  least  for  a  sample  of  "graduated"  learners. 
Further,  it  must  be  able  to  compare  those  measures  with  the 
end-of-training  tests  administered  to  the  same  learners,  to 
identify  any  deficiencies  in  the  test  specifications.  The  test 
specifications  cannot  be  considered  to  be  "complete"  if  the 
ability  to  correlate  test  results  with  real  job  performance  is  not 
built  into  the  Design  subsystem. 

Conclusion:  This  taxon  is  relevant  to  the  application. 
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APPENDIX  B. 


Application  of  the  APM  (Measurement) 


to  the 


raimn 


Appendix  B  supports  the  textual  material  presented  in  Section  III  of  this 
report.  It  contains  Figures  B1  through  BIO.  Briefly,  Figures  B1  through  B6 
present  all  of  the  Army  Handgun  Training  Design  Subsystem  Potentialities 
Requirements.  Figure  B7  illustrates  a  sample  AHTDS  Performance  Process 
Requirements,  and  Figure  B8  illustrates  an  example  of  the  process  design 
specifications  for  the  Figure  B7  requirements.  Similarly,  Figure  B9  illustrates 
a  sample  AHTDS  Performance  Product  Requirements,  and  Figure  BIO  illustrates 
an  example  of  the  product  design  specifications  for  the  Figure  B9 
requirements. 
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Performance  Potentiality  Hierarchy  Number  1:  The  capability  of  Identifying  training  goals  and  priontic* 


TKc  Handgun  Training  Design  Subayatem 
mu  it  poiacai  the  ability  to  Identify  train¬ 
ing  goali  and  prlorttiei  for  all  intended 
military  uaer*  and  usei  of  the  handgun. 


1.1.1 

Civen  a  Hat  of  handgun  uacra 
and  uaaa.  the  Oeaign  Subsystem 
muat  be  able  to  define  the 
acope  of  learning  needed  for 
each  handgun  ueer/uae  in  term* 
of  the  knoeledge.  akilli.  and 
attitudee  required . 


1. 1.1.1 

The  Handgun  Training 
Deeign  Subiyitetn  muat 
demonstrate  that  It  poe- 
aeaaei  the  ability  to 
Identify,  for  eech  hand¬ 
gun  ueer/uae,  exactly 
what  the  uaer  muat  know; 
maactly  what  ekilla  the 
uaer  muat  poeaeaa;  and 
exactly  what  attitudea 
the  ueer  muet  manifeat. 

1.1.12 

TFeTTandgun  Training 
Dealgn  Subayatem  muat 
demonatrate  that  It  poa- 
eeaaea  the  ability  to 
analyte  the  identified 
knowledge,  akilli,  and 
attitudea  to  determine 
their  auitability  for 
training. 


1.1-2 

Given  a  Hat  of  handgun  uaera 
and  uaea,  the  Dcatgn  Subayatem 
muit  be  able  to  define  what 
each  uaer  ultloialely  will  be 
able  to  dn  with  the  handgun. 

In  training  goal  term*. 


1,1, 2.1 

The  Handgun  Training 
Design  Subayatem  muat 
demonatrate  that  it  poa- 
eeaaea  the  ability  to 
define  the  neceaaary 
levela  of  performance 
achieveewnta  that  each 
handgun  uaer  muat  mani¬ 
feat  In  order  to  apply 
the  handgun  to  Ita 
Intended  uaea. 

1.1 .2.2 

The  Handgun  Training 
Deaign  Subayatem  muat 
demonatrate  that  it  la 
able  to  determine  cor¬ 
rectly  the  pre-training 
levela  of  the  relevant 
performance  achievement! 
poeeeaaed  by  member!  of 
the  Intended  handgun  uaer 
population*. 

1.1. 2.1 

The  Handgun  Training 
Deaign  Subayatem  muat 
demonatrate  that  it  poa- 
aaaaea  the  ability  to 
dtaeem  correctly  the 
diacrepanclea  exlatlng 
between  current  and  re¬ 
quired  levela  of  relevant 
performance  achievement* 
within  the  Intended  hand¬ 
gun  uaer  population!. 


1.1.1 

fclven  a  Hat  of  handgun  uaera 
and  uaea,  and  given  the  train¬ 
ing  goal*  defined  for  each 
ueer/uae.  the  Dedgn  Subayatem 
muat  be  able  to  ,dentlfy  the 
relative  Importance  of  each 
defined  goal. 


1.1 .1.1 

The  Handgun  Training 
Deaign  Subayatem  muet 
demonatrate  that  it  haa 
the  ability  to  formulate 
eaeeaamenta  of  training 
goal  importance. 


1.1. 3. 2 

The  Handgun  Training 
Deaign  Subayatem  muat 
demonatrate  that  It  can 
devise  or  oelect  a  suit¬ 
able  scheme  for  ranking 
th'  relative  Importance 
of  Ita  training  goal*. 


1.1. t 

Given  a  Hat  of  handgun  uaera 
and  uaea,  and  given  the  trair  - 
ing  goals  defined  for  each 
uaer/use,  th*  Deaign  Subsystem 
muat  be  able  to  express  the 
goal*  in  behavioraliy  oriented 
terms  to  establish  a  basis  for 
Identifying  handgun  uaer  per¬ 
formance  objectives. 

1. 1.4.1 

The  Handgun  Training 
Design  Subayatem  must 
demonatrate  that  it  can 
devise  or  oelect  a  suit¬ 
able  taxonomy  of  goals/ 
objectives  to  be  used  to 
classify  ita  intended 
training  outcomes. 


1.1  ■■4.2 

The  Handgun  Training 
Deaign  Subsystem  must 
demonatrata  that  it  is 
capable  of  stating  Ita 
goals  In  termi  that  per¬ 
mit  the  necessary  user 
behaviors  to  be  identified 


Figure  B-L  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Potentialities  Needed:  Hierarchy  Number  1 
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Performance  Potentiality  Hierarchy  Number  2: 

The  capability  of  cslabTiahins  performance  ofejectivct 


1.2 

TKf  Handgun  Training  D«aign  Subayatcm 
must  possess  the  ability  to  eatabliah 
performance  objectives  for  all  intended 
military  uaera  and  uaea  of  the  handgun. 


1.2.1 

Civen  a  list  of  handgun  uaera  and 
uaea,  the  Deaign  Subayatcm  must 
be  able  to  state  in  perfortnanee- 
oriented  terma  what  the  handgun 
users  will  be  able  to  perform  as 
a  result  of  training. 


1.2. 1.1 

The  Handgun  Training  Deaign 
Subsystem  must  demonstrate 
that  It  possesses  the  abil¬ 
ity  to  define  all  of  the 
tasks  that  the  handgun  users 
win  "be  expected  to  perform. 


1.2. 1.2 

The  Handgun  Training  Design 
Subsystem  must  demonstrate 
that  it  possesses  the  abil¬ 
ity  to  define  all  of  the 
conditions /circumstances 
wider  which  each  of  the 
tasks  is  to  be  performed. 


1. 2.1.1 

The  Handgun  Training  Design 
Subsystem  must  demonstrate 
that  It  poo  sc  s  sci  the  abil¬ 
ity  to  define  all  of  the 
tralningfpsrfortuance  Stand¬ 
ard*  t or  each  task  and  for 
all  of  the  conditions  under 
which  the  task  la  to  be 
performed  in  terms  of  mini¬ 
mum  acceptable  proficiency. 


TSTvan  a  list  of  handgun  uaera  and 
uses  and  given  the  stated  perfor¬ 
mance  objectives,  the  Design  Sub¬ 
system  must  be  able  to  dafine  a 
scheme  for  objectively  assessing 
handgun  user  performance  prior  to 
and  after  receiving  the  designed 
training. 


1.2. 21 

The  Handgun  Training  Deaign 
Subsystem  must  demonstrate 
that  it  possesses  tht  abil¬ 
ity  to  develop  pre-teat  Items 
to  determina  handgun  users' 
existing  knowledges,  skills 
and  attitudes. 


1.2. 2. 2 


The  Handgun  Training  Design 
Subsystem  must  demonstrate 
that  It  possesses  the  abil¬ 
ity  to  develop  post -teat 
Items  to  determine  handgun 
uaera'  post-training  knowl¬ 
edges,  skills  end  attitudes. 


Figure  B-2.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Potentialities  Needed:  Hierarchy  Number  2 


Figure  B-3.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Potentialities  Needed:  Hierarchy  Number  3 


Performance  Potentiality  Hierarchy  Number  4:  The  capability  of  defining  the  training  content 


1.4 

TTic  Handgun  Training  Design  Subsystem 
muat  poaaeaa  the  ability  to  define 
the  handgun  training  content  required 
to  achieve  the  uaera'  performance 
objectivea. 


Given  •  Hat  of  handgun  uaera  and 
■aea,  and  given  the  defined  training 
goale  and  analyaed  performance 
objectivea,  the  Deaign  Subayetem 
muat  be  able  to  aelect  atlmuli  for 
each  performance  objective  relevant 
to  the  domains  of  learning. 


The  Handgun  Training  Design 
Subsystem  must  demonstrate 
that  It  possesses  the  abil  ¬ 
ity  to  define  procedures, 
facta,  names,  etc.,  that  are 
relevant  to  information-type 
objectives. 


The  Handgun  Training  Design 
Subsystem  must  demonstrate 
that  it  possesses  the  abil¬ 
ity  to  define  concepts, 
rules,  algorithms,  etc., 
relevant  to  mental  shills 
objectives. 


The  Handgun  Training  Design 
Subsystem  must  demonstrate 
that  It  possesses  the  abil¬ 
ity  to  define  handgun  user/ 
■see  movements,  actions, 
timings,  etc.,  for  relevant 
physical  skill  objectives. 


1.4. 1.4 


The  Handgun  Training  Desig 
Subsystem  must  demonstrate 
that  it  possesses  the  sbil- 
*'r  define  handg  seer' 


s'-a  choices,  values,  etc., 
relevent  to  attitude  objec¬ 
tives. 


n 


1.4_J 

Given  s  list  of  handgun  users  and 
uses,  and  given  the  defined  training 
goals  and  snalytsd  performance 
objectives,  the  Design  Subsystem 
■uit  be  able  to  select  en  appro¬ 
priate  inetructional  eetting  for 
each  performance  objective. 


1.4.2. 1 

The  Handgun  Training  Design 
Subsystem  must  demonstrate 
that  it  possesses  the  abil¬ 
ity  to  assess  the  applicability 
of  Job  Performance  Aids  (JPAs) 
to  enhance  the  learning 
process . 


1.4,1 

Given  a  list  of  handgun  users  and 
uses,  and  given  the  defined  tram:- 
goals  and  analysed  performance 
objectives,  the  Design  Subsystem 
mutt  be  able  to  define  an  mstrur 
t  Ion  si  sequence  which  provides  for 
the  optimum  transition  and  suppn*  r 
from  one  performance  obtective  : -- 
another. 


The  handgun  Training  Design 
Subsysten  must  demonstrate 
that  it  possesses  the-  abil¬ 
ity  to  eoquence  the  training 
relative  to  the  objectives 
dependency  order  nf  required 
■kills,  knowledges  and 
attitudes . 


1,4. 2.2 

The  Handgun  Training  Design 
Subsystsm  must  demonstrate 
that  It  possesses  the  abil¬ 
ity  to  assess  the  applicability 
of  Self-Tsaching  Exportable 
Packages  ISTEPtl  a*  a  vehicle 
to  enhance  the  learning 
process. 


The  Handgun  Training  Design 
Subsvstem  must  demonstrate 
ihss  it  poetesses  the  abil¬ 
ity  to  assess  ths  applicability 
of  formal  on-the-job  training 
(OJT) . 


The  handgun  Training  Design 
Subsystem  must  demonstrate 
that  It  possesses  the  abil¬ 
ity  to  sequence  the  training 
based  upon  the  supportive 
order  of  knowledges,  skiit*. 
and  attitudes  from  one  per¬ 
formance  elective  to  other 
performance  objectives. 


The  Handgun  Training  Deaign 
Subeystem  must  demonstrate 
that  it  possesses  ths  abil¬ 
ity  to  assrss  the  applicability 
of  Installation  Support  Schools 
(1SS)  as  a  possible  instruc¬ 
tional  setting. 


The  Handgun  Training  Design 
Subsystem  must  demonstrate 
(hat  It  possesses  the  abil¬ 
ity  to  assess  ths  applicability 
Of  Resident  Schools  (RS)  as  a 
possible  Instructional  wetting. 


Figure  B-4.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Potentialities  Needed:  Hierarchy  Number  4 


Performance  Potentiality  Hierarchy  Number  3:  The  capability  of  defining  training  procedure. 


l.S 

TKe  Handgun  Training  Design  Subsystem 
must  possess  the  ability  to  define 
the  training  procedures  to  be  employed 
for  all  intended  military  users  and 
•sea  of  the  handgun. 


Given  a  list  of  handgun  users 
and  uses,  and  given  the  train¬ 
ing  goals  and  priorities,  ana¬ 
lysed  performance  objectives 
and  defined  training  content, 
specify  the  training  prepara¬ 
tion  procedures  required  to 
assure  optimum  learning. 


TKeTTandgun  Training 
Design  Subsystem  must 
demonstrate  that  it  pos¬ 
sesses  the  ability  to 
define  required  teacher 
preparation  activities  to 
assure  optimum  learning. 


The  Handgun  Training 
Design  Subsystem  must 
demonstrate  that  it  pos¬ 
sesses  the  ability  to 
define  facilities  prepa¬ 
ration  requirements  to 
accomplish  handgun 
training  objectives. 


Given  a  list  of  handgun  users 
and  uses,  and  given  the  train¬ 
ing  goals  and  priorities,  ana¬ 
lysed  performance  objectives, 
and  defined  training  content, 
specify  the  training  presenta¬ 
tion  proesdures  required  to 
assure  optimum  learning. 


l.S. 2.1 

The  handgun  Training 
Design  Subsystem  must 
demonstrate  that  it  pos¬ 
sesses  the  ability  to 
define  presentation 
■wdia/media  alternatives 
to  assure  optimum 
learning . 


The- Handgun  Training 
Design  Subsystem  must 
demonstrate  that  it  pos¬ 
sesses  the  ability  to 
define  presentation  pro¬ 
cedures  and  requirements 
for  placing  emphasis  to 
enhance  the  need-to-Iearn. 


l.S.  3 

Given  a  list  of  handgun  users 
and  uses,  and  given  the  train¬ 
ing  goals  and  priorities,  ana- 
lyied  performance  objectives, 
and  defined  training  content, 
specify  the  application  pro- 
cedurss  required  to  assure 
optimum  trainee  achievement  of 
performance  objectives. 


The  Handgun  Training 
Design  Subsystem  must 
demonstrate  that  it  pos¬ 
sesses  the  ability  to 
define  training  applica¬ 
tion  procedures  for  en¬ 
hancing  the  handgun 
learner’s  involvement  or 
participation  In  the 
handgun  training. 


The  Handgun  Training 
Design  Subsystem  must 
demonstrate  that  It  pos¬ 
sesses  the  abilitv  to 
define  procedures  which 
will  enhance/stimulate 
learner  performance. 


a  list  of  handgun  u»rrs 
and  uses,  and  giver  ana'.r* 
performance  objective*  art? 
defined  training  content, 
■pecifv  evaluation  procec4  u'*' 
to  aasurt  achievement  of 
formance  objective*. 


M.i.l 

The  Han  dgun  Training 
Deaign  Subsystem  fre-v* 
demonstrate  that  it  po« 
•esse*  the  ability  to 
define  procedures  to 
the  learner'e  achievers  • 
of  Information,  mental 
•kills,  physical  skill*  a»>  ) 
attitude  objectives,  i.e 
the  domains  of  learning 


The  Handgun  Training 
Deaign  Subsystem  mu«t 
drmonstrsir  thst  It  pc* 
srssrs  the  sbihtv  to 
drfine  procedures  for 
assessing  appropriate 
handgun  learner  bcha-.  t*  - 
action. 


l.S. l.S 

The  handgun  Training 
Design  Subsystem  must 
demonstrate  thst  it  pos¬ 
sesses  the  ability  to 
defins  handgun  learner 
attention -gaining 
procedures. 


l.S. 2. 3 

The  Handgun  Training 
Design  Subsystem  must 
demonstrate  that  it  pos¬ 
sesses  the  ability  to 
define  learning  pro¬ 
cedures  to  enhance  hand¬ 
gun  learner's  comprehen¬ 
sion  and  retention. 


l.S ,3.1 

The  Handgun  Training 
Design  Subsystem  must 
demonstrate  thst  It  pos¬ 
sesses  the  ability  to 
define  procedures  that 
will  provide  learning 
guidance  to  handgun 
Warners. 


The  handgun  Training 
Design  Subsystem  must 
demonstrate  that  It  pcs 
eesses  the  ability  to 
define  handgun  perfor¬ 
mance  objeettve- 
tnforsdng  procedures. 


Tbs  handgun  Training 
Design  Subsystem  must 
demonstrate  that  It  pos¬ 
sesses  the  ability  to 
define  proesdures  for 
stimulating  handgun 
Warner's  recall  of 
handgun  training  pre¬ 
requisites. 


l.S. 3. 4 


Tha  Handgun  Training 
Design  Subsystem  must 
demon  strste  thst  It  pos¬ 
sesses  ths  ability  to 
define  procedures  which 
trill  assure  that  Warning 
feedback  is  provided  to 
handgun  Warners. 


Ths  handgun  Training 
Design  Subsystem  swat 
demonstrate  that  it  pos¬ 
ses  sea  the  ability  to 
define  proesdures  which 
assure  thst  adequate/ 
sufficient  practice  has 
been  accomplished  to 
consistently  achieve 
performance  objectives. 


l.S. 1.1 

The  Handgun  Tre  nr'  > 
Design  Subsystem  «.c* 
demonstrate  thst  it  .  r  • 
aesses  the  ability  ■ 
define  procedure*  tha' 
will  assure  reh-hlr 
assessment  of  hen. I  j 
Warner  achievement  ->f 
performance  objective*. 


Figure  B-5.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Potentialities  Needed:  Hierarchy  Number  5 


B-6 


Performance  Potentiality  Hierarchy  Number  6: 
~TKT  capability  of  evaluating  the  curriculum 


1.6 

The  Handgun  Training  Design  Subsystem 
Bust  possess  the  ability  to  evaluate 
the  curriculum  developed  for  intended 
Army  users  and  uses  of  the  handgun. 


1.6.1 

Given  a  list  of  handgun  users  and 
uses,  as  well  as  defined  handgun 
training  system  goals  and  priori¬ 
ties,  defined  performance  objec¬ 
tives.  defined  training  content 
and  training  procedures,  the 
Design  Subsystem  Bust  be  able  to 
validate  and/or  revise  the  bases 
for  developing  the  handgun  train¬ 
ing  system  curriculum. 


1.6.2 

Giver  s  list  of  handgun  users  and 
uses,  as  well  as  defined  handgun 
training  aystem  goals  and  priori¬ 
ties.  defined  performance  objec¬ 
tives,  defined  training  content 
and  training  procedures,  the 
Design  Subsystem  must  be  able  to 
validate /revise  the  developing 
handgun  curriculum  itself. 


The  Handgun  Training  Design 
Subsystem  must  demonstrete 
thst  it  possesses  the  sbil- 
ity  to  conduct  entry-level 
testing  of  reprcscntstlve 
handgun  laamers. 


1.6. 2.1 


The  handgun  Training  Desig 
Subsystem  must  demonstrate 
that  it  possesses  the  sbil- 
Ity  to  obtain  error  data 
related  to  trial  learner 
training . 


n 


The  Handgun  Training  Design 
Subsystem  Bust  demonstrate 
that  it  possesses  the  abil¬ 
ity  to  conduct  post-training 
testing  of  representative 
handgun  users. 


1.6. 2. 2 


The- Handgun  Training  Design 
Subsystem  must  demonstrate 
thst  it  possesses  ths  abil¬ 
ity  to  cbtsin  curriculum 
design  feedback  from  poten¬ 
tial  learners /job  incumbents. 


The  handgun  Training  Design 
Subsystem  Bust  demonstrate 
that  it  possesses  the  abil¬ 
ity  to  correlate  post-test 
training  data  with  subse¬ 
quent  learners'  job  perfor¬ 
mance. 


Figure  B-6.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Potentialities  Needed:  Hierarchy  Number  6 


Performance  Procwxi  Hierarchy  Number  >:  An  approach  for  «n»ltilm  tasks  nlttttd  for  trelnlni 


2.3 

The  Handgun  Training  Design  Subsystem 
Bust  be  able  to  plan  an  approach  (or 
analyilng  handgun  uaar  tasks  which 
ar«  aalccted  for  training. 


2.3.1 

TTSHting  the  job  analytia  of  hand¬ 
gun  uaar  a  and  uaoa  and  an  aasesa- 
aent  of  thooa  handgun  uaara'  taska 
aa  lac  tad  for  training  development , 
the  Design  Subayatem  mat  ba  able 
to  plan  an  approach  to  identify 
the  information  processing 
entailed  in  each  of  the  taeks 
Selected  for  training. 


2.3.2 

TTCTTting  the  job  analyaia  of  hand¬ 
gun  users  and  uaea  and  an  assess¬ 
ment  of  those  handgun  users'  tasks 
selected  for  training  development, 
the  Deaign  Subayatem  must  be  able 
to  plan  an  approach  to  classify 
the  tasks  selected  for  training  in 
terms  of  the  domains  of  learning 
involved. 


2.3.3 

Oilliting  the  job  analysis  of  hand¬ 
gun  users  and  uses  and  the  assess¬ 
ment  of  those  handgun  users'  tasks 
selected  for  training  development, 
the  Deaign  Subsystem  muat  be  able 
to  plan  an  approach  to  identify 
the  conditiona  for  learning  each 
of  the  tasks  selected  for  training. 


2.3. 1.1 


The  Handgun  Training  Design 
Subsystem  must  be  able  to 
plan  an  approach  to  analyte 
task  decision  flows  to  reduce 


processing  to  a  sequence  of 
binary  decisions. 


2.3.2. 1 

Yhe  Handgun  Training  Design 
Subsystem  muat  be  able  to 
plan  an  approach  to  identify 
knowledges  which  muet  be 
acquired. 


2. 3. 3.1 

The  Handgun  Training  Design 
Subsystem  muat  be  able  to 
plan  an  approach  to  analyse 
Information -proeessin  g 
requirements  to  Identify  pre¬ 
requisite  knowledge  eepected 
of  learners. 


2. 3 .1.2 

TS-s  »Tandgun  Training  Design 
Subsystem  muat  be  able  to 
plan  an  approach  to  analyte 
the  task  decision  levels  in 
terms  of  nature,  complexity, 
and  time. 


2.3.2 1.2 

The  Handgun  Training  Design 
Subsystem  must  be  able  to 
plan  an  approach  to  Identify 
the  mental  skills  which 
must  be  acquired. 


2.3. 3.2 


The  Handgun  Training  Desig 
Subsystem  must  be  able  to 
plan  an  approach  to  analyte 
task  requirements  to  iden¬ 
tify  prerequisite  eienta) 
skills. 


n 


2.3.2. 3 

The  Handgun  Training  Design 
Subsystem  must  be  able  to  plan 
an  approach  to  Identify  the 
physical  skills  which  mutt  be 
acquired. 


2.3. 3.1 

The  Handgun  Training  Design 
Subsystem  muet  be  able  to  plan 
an  approach  to  analyte  the 
physical  requirements  to  iden¬ 
tify  prerequisite  motor  skills. 


2. 3. 2. 4 


The  Handgun  Training  Design 
Subsystem  must  be  able  to  plan 
an  approach  to  identify  atti¬ 
tudes  to  be  acquired. 


2. 3.3.4 

The  Handgun  Training  Design 
Subsystem  swat  be  able  to  plan 
an  approach  to  analyte  the 
attitudina!  requirements  to 
identify  expected  prerequisite 
learner  attitudes. 


Figure  B-7.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Processes  Needed:  Hierarchy  Number  3 


Process  Performance  Requirement  Number  3: 

An  approach  for  analyzing  tasks  selected  for  training 


(2.3) 


The  AHT  Command  Subsystem  will  have  reviewed  as  part  of  the 
curriculum  design  effort,  the  handgun  user's  job  analysis  and  the 
analysis  to  determine  which  tasks  should  be  selected  for  training 

development.  The  Design  Subsystem  will  closely  interact  with  the 
Command  Subsystem  in  determining  which  of  the  tasks  considered  for 
training  will  actually  be  selected  for  training  development.  The  final 
determination  of  tasks  selected  for  training  will  be  based  on  such  factors 
as  task  training  costs,  available  time  and  resources  as  well  as  tasks  which 
may  be  already  learned  through  previous  training  and  consequently  may 
be  considered  as  a  prerequisite  to  the  AHT.  Once  the  Design  Subsystem 
has  received  the  approved  list  of  tasks  to  be  trained  they  will  be 

analyzed  utilizing  operational  sequence  diagram  (OSD)  techniques  and 
decision  trees  which  will  describe  in  detail  the  learning  hieiarchy 

associated  with  each  task. 

(2.3.1,  2.3.2,  2.3.3) 

The  team  will  examine  the  information  processing  entailed  in  each  of 
the  tasks  selected  for  training  in  the  context  of  representative  scenarios 
designed  specifically  in  the  use  of  the  handgun  for  personal  defense 

actions.  It  is  anticipated  that  the  Command  Subsystem  will  supply 
realistic  scenarios  in  such  detail  to  provide  the  analysts  with  information 
which  describes  the  conditions  and  performance  environment  in  which  the 
user  must  utilize  the  handgun  for  self  defense  purposes.  The  sequential 
operations  in  performing  each  of  the  tasks  will  be  analyzed  via  an  OSD 
technique  until  all  of  the  task  elements  and  their  required  sub-elements 
have  been  defined.  The  analysis  will  identify  all  the  handgun  user  task 
inputs,  the  processing  required,  the  resulting  outputs,  as  well  as  task 
element  inter-relationships.  At  each  level  of  task  element  analysis,  the 
analyst  will  determine  what  information  the  learner  must  know,  recall  or 
state  to  perform  the  task.  The  analysis  will  identify  categories  of 
information  which  the  potential  handgun  learner  must  process.  These 
analyses  also  will  be  the  basis  for  identifying  prerequisite  abilities  that 
the  handgun  learner  will  bring  to  the  training  situation,  as  well  as  for 
identifying  the  learning  the  AHT  must  provide  as  described  below. 


Figure  B-8.  Sample  Design  Specifcation  of  a 
Process  Aspect  of  the  AHTDS 

(In  response  to  the  Performance  Requirements  illustrated  in  Figure  B-7.) 
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(2. 3. 1.1,  2. 3. 1.2) 

The  sequential  analyses  of  tasks  will  identify  handgun  user  decision 
flows.  These  decision  flows  will  be  further  analyzed  utilizing  a  binary 
decision  tree  (yes/no)  technique.  The  decision  flow  analysis  will  include 
an  appropriate  time  line  as  well  as  columns  to  identify  the  nature  of 
decisions  and  to  classify  the  complexity  of  the  decision  levels  from  simple 
to  complex  using  a  scale  of  1  to  5.  The  complexity  of  these  decisions  will 
be  based  on  such  factors  as  criticality  of  the  decision  with  regard  to  task 
completion  delay  tolerance  and  number  of  factors  influencing  the  yes /no 
branches  of  the  decision  tree. 

(2. 3. 2.1,  2. 3. 2. 2,  2. 3. 2. 3,  2. 3. 2. 4) 

The  information-decision-action  attribute  model,  described  in  detail  in 
Section  **  of  this  specification,  will  be  the  basis  for  performing  the 
analyses  to  classify  the  tasks  selected  for  training  in  terms  of  the 
domains  of  learning  involved.  An  analysis  of  each  of  the  tasks  will  be 
performed  to  develop  categories  of  information  the  potential  handgun 
learner  must  possess  to  complete  tasks/task  elements.  The  analysis  will 
detail  the  bodies  of  knowledge  the  learners  must  be  able  to  recall  or 
state.  To  analyze  mental  skills  required,  the  task  elements  will  be 
analyzed  in  detail  to  determine  rule  learning  and  using,  pattern 
recognition,  symbol  identification,  detection  cues  and  decision  making. 
The  team  of  analysts  will  prepare  a  detailed  hierarchial  documentation  of 
the  mental  skills  a  potential  learner  must  possess  in  the  performance  of 
tasks.  To  analyze  physical  skills  required,  again  each  task  element  will 
be  sequentially  analyzed  to  determine  the  sequence  of  physical  (motor) 
skills  required,  a  hierarchy  of  the  sequence  of  physical  skills  required  to 
perform  required  tasks  will  be  fully  documented.  An  analyses  will  also  be 
performed  to  determine  the  attitudes  a  handgun  learner  should  possess  in 
performing  the  handgun  systems  functions  i.e.,  serve  as  a  short-term 
persona!  defense  weapons  system.  This  analysis  will  involve  some 
speculative  thinking  since  it  is  not  always  possible  to  observe  attitudes 
directly.  The  team  will  analyze  each  task/task  element  and  ask  the 
question,  "What  would  a  potential  handgun  learner  do  if  he  had  the 
desired  attitude?"  The  team  will  use  a  consensus  approach  to  develop  a 
list  of  the  attitudes  potential  learners  must  possess  in  order  to  exhibit 
the  desired  attitudes. 

(2. 3. 3.1,  2. 3. 3. 2,  2. 3. 3. 3,  2. 3. 3. 4) 

The  analyses  to  determine  potential  learner  prerequisite  knowledge, 
mental  skills,  physical  skills,  and  attitudes  will  be  accomplished  utilizing 
the  outputs  of  the  analyses  described  in  the  previous  paragraph  and  the 
analyses  performed  in  Section  **  which  identifies  entry-level  behaviors 
that  the  potential  learner  brings  to  the  learning  situation.  A  comparison 
of  entry  level  behaviors/attributes  that  a  learner  brings  to  the  training 
situation  and  those  behaviors/attributes  that  a  trained  handgun  user  must 
exhibit  will  define  those  knowledges,  mental  and  physical  skills  as  well  as 
attitudes  the  learner  must  acquire  via  the  AHT.  The  results  of  these 
analyses  will  be  documented  for  each  of  the  domains  of  learning. 

Figure  B-8  (Concluded) 
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Performance  Product*  Hierarchy  Number  3:  The  capability 


The  Handgun  Training  Deaign  Subayatem 
■uat  be  able  to  produce  leaaon  plana 
lo  eupport  intended  handgun  uaer 
training  object  I  vea. 


3.3.1 

btiiiaing  the  developed  handgun  «eer 
Job  anaJyaia  and  performance  objec¬ 
tive*,  the  Deaign  Subayatem  emat 
provide  eoaaplete  epecifieation  of 
the  inelructional  content  for  each 
training  objective. 

1.3. 1.1 

flu  handgun  Training  Deaign 
Subayatem  must  produce  epecifi- 
catkma  of  each  handgun  uaer 
training  objective. 

1.3. 1.2 

The  Handgun  Training  Deaign 
Subayatem  muat  produce  apecifi- 
cationa  of  aaaunied  handgun 
laamar  prerequialtea. 

3.3.1 .3 

The  Handgun  Training  Deelgn 
Subayatem  muat  produce  an 
itemiiatian  of  Information 
topica. 


3.3 ,  j  4 

1  he  Handgun  Training  Deaign 
Subayatem  muat  produce  an 
Itcmiiatlon  of  mental  akill 
otepa. 


3.3. 1.3 

The  Handgun  Training  Deelgn 
Subayatem  cruet  produce  an 
Hemiaation  of  phyeical  Ull 

otapa. 


3.3. 1.4 

The  Handgun  Training  Deaign 
Subayatem  muat  produce  an 
Itemiiatian  of  learner  alti¬ 
tudinal  component! . 

S. 3.1.7 

The  handgun  Training  Daeign 
Subayatem  enact  produce  an 
Itead action  of  practice  eaer- 


The  Handgun  Training  Daeign 
Subayatem  Met  produce  an 
Hemiaation  ef  teat  p ruble me/ 


3.3.2 

TOKIlng  the  developed  handgun  uaer 
job  analyaia  and  performance  objec- 
tlvei,  the  Deaign  Subayatem  muat 
provide  comilete  epecifieation  of 
the  inatrur  Iona)  procedural  for 
each  training  objective. 

3. 3.2.1 

The  handgun  Training  Deaign 
Subayatem  muat  produce  a  pacifi¬ 
cation  a  of  procedural  for  pre¬ 
paring  handgun  karneri  to 
achieve  each  training  objective. 

3. 3 .2. 2 

The  Handgun  Training  Deaign 
Subayatem  muat  produce  epeeifl- 
catlone  of  exactly  hoo  the 
tnatructiona)  content  topica / 
otepa  aril)  be  preaented. 

3. 3. 2. 3 

The  Handgun  Training  Deaign 
Subayatem  must  produce  apecifi- 
cationa  of  exactly  hoar  the 
handgun  learner*  mill  practice/ 
apply  the  inatructional  content. 

3.3,2  4 

The  Handgun  Training  Deaign 
Subayatem  muat  produce  apecifi- 
cation*  of  exactly  how  the 
handgun  learner*  oil!  be  teeted 
on  the  inatructional  content. 

3.3.2 j 

The  handgun  Training  Deaign 
Subayatem  mual  product  ape ci fl¬ 
eet  ion  a  of  procedure*  for  deal¬ 
ing  with  handgun  learner*  who 
fail  to  dempnatrata  achievement 
of  any  of  the  training  objec¬ 
tive*. 


3.3.3 

lltiliting  the  developed  handgun  uaer 
job  analyaia  and  performance  objec¬ 
tive*.  the  Deaign  Subayatem  muat 
provide  complete  epecificttion  of 
the  inatructional  cireumattnee*  for 
each  training  objective. 

3.3,3  1 

The  Handgun  Training  Deaign 
Subayatem  muat  product  aperifl- 
catton*  of  tho  location  at  which 
each  inatructional  event  will 

take  place. 

3. 3. 3 i.2 

The  Handgun  Training  Deaign 
Subayatem  muat  produce  specifi¬ 
cation*  of  the  equipment  needed 
to  eupport  the  training. 


3. 3. 3. 3 

The  Handgun  Training  Deaign 
Subayatem  muat  produce  apecifi- 
cationa  of  apodal  location  I 
equipment  aet-upfconfiguration 
requirement* . 

l.I.M 

The  Handgun  Training  Deaign 
5ubay*tem  muat  produce  apreifi- 
cationa  of  inatructional  per¬ 
sonnel  needed.  ' 


3.1. 3.3 

The  Handgun  Training  Design 
Subsystem  must  produce  specifi¬ 
cations  of  time  and  schedule 
requirements. 


Figure  B-9.  Army  Handgun  Training  Design  Subsystem  Performance  Requirements 
Basic  Products  Needed:  Hierarchy  Number  3 
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Product  Performance  Requirement  Number  3: 
The  capability  of  producing  lesson  plans. 

(3.3) 


The  team  recognizes  that  the  development  of  lesson  plans  is  one  of 
the  most  important  requirements  in  the  training  development  process.  It 
is  at  this  point  that  all  of  the  preceding  analyses  will  be  utilized  to 
produce  the  actual  instructional  products,  i.e.,  lesson  plans  which 
support  the  AHT  objectives.  The  team  that  has  been  assembled  are  well 
versed  in  producing  lesson  plans  to  meet  these  objectives.  The  analyses 
to  develop  the  handgun  user's  job  analysis  and  performance  objectives  will 
be  the  basis  for  lesson  plan  development.  These  analyses  include  the 
following  information: 


Training  objectives 
Tasks  to  be  performed 
Conditions  of  performance 
Training  standard  of 
acceptable  performance 
Test  items  for  all  objectives 
Reference  to  the  task/task 
element  from  which  the 
training  objective  was 
derived 


Training  objective  sequence 
number  and  how 
objectives  were  structured 
Learning  category  and 
sub-category  for  each  training 
objective 

Learning  activities  for  each 
objective 

Planning  information  from  other 
AHT  subsystems 
Existing  handgun  instructional 
programs 


The  team  will  use  Field  Manual  21-6  "Military  Training"  as  a  guide  to 
produce  lesson  plans  which  provide  practical,  economical  aid  in  preparing 
and  delivering  the  training.  The  lesson  plans  will  not  be  designed  to 
record  every  word  of  the  instructor's  presentation.  The  following 
minimum  information  will  be  included  in  all  of  the  produced  lesson  plans: 


The  Training  Objective(s) 

All  Intermediate  Training 
Objectives  (if  any)  listed 
in  the  sequence  to  be  taught 
Administrative  instructions 
(as  required) 

When  the  training  will  be 
conducted 
Training  location 
Who  will  be  trained 


-  Principal  and  assistant 
trainers 

-  Training  aids,  devices  and 
equipment  to  be  used 

-  References 

Training  sequence  and  time 
estimate 

Safety  restrictions 
Additional  information  required 
by  local  SOPs 


Figure  B-10.  Sample  Design  Specification  of  a 
Product  Aspect  of  the  AHTDS. 

n  response  to  the  Performance  Requirements  illustrated  in  Figure  B-9.) 


(3.3.1,  3.3.2,  3.3.3) 


In  providing  complete  specifications  of  the  instructional  content,  the 
team  will  prepare  a  content  outline  to  satisfy  each  training  objective. 
Each  training  objective  will  contain  the  tasks  which  must  be  learned,  the 
conditions  of  task  performance  and  the  task  standard  of  acceptable 
performance.  The  specifications  will  include  learner  prerequisites  as  well 
as  the  inti  rim  objectives  which  must  be  performed  to  achieve  the 
objective.  The  outline  will  itemize  the  information  topics,  mental  and 
physical  skills,  and  attitudes  which  must  be  acquired  as  a  result  of 
learning.  The  lesson  plans  will  include  practical  exercises  and  test  items 
to  stimulate  learning  as  well  as  providing  the  instructors  with  criteria  for 
evaluating  trainee  learning  relative  to  the  training  objective. 
Instructional  strategies  will  be  included  for  each  lesson  plan.  They  will 
be  in  the  form  of  guidelines  and  will  offer  strategies  which  will  aid  the 
instructor  in  presenting  the  material.  The  strategies  will  suggest  ways  to 
keep  the  learners  involved.  They  will  cue  the  instructor  as  to  when 
instructional  aids  are  appropriate  as  well  as  when  to  emphasize /repeat 
instructional  material.  The  specifications  for  instructional  circumstances 
will  be  detailed  in  the  administrative  instructions  of  the  lesson  plan  as 
well  as  in  the  POI  course  guide. 

(3. 3. 1.1  -  3.3. 1.8) 

The  Task  Inventory  developed  in  Section  **  of  these  specifications, 
will  be  used  as  the  basis  for  deriving  the  specific  terminal  and 
intermediate  Training  Objectives,  Each  Training  Objective  will  be  clearly 
stated  and  will  consist  of  the  following  three  elements: 

°  Tasks  -  What  knowledges/skills  the  handgun  learners  must 
acquire 

°  Conditions  -  Under  what  conditions  the  learners  must  acquire  the 
knowledges/ skills 

°  Training  standard  -  A  description  of  minimum  acceptable 
performance 

The  various  kinds  of  learned  capabilities  have  various  kinds  of 
prerequisites — previously  learned  entities  which  are  available  from  the 
learners  memory  at  the  time  new  learning  begins.  A  notation  of  the  type 
of  prerequisite  required  for,  or  helpful  to,  the  learning  appropriate  to 
each  listed  Training  Objective  will  be  identified.  For  each  category  of 
learned  capability,  the  following  will  be  considered: 


Figure  B-10  (Continued) 
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°  Mental  skill  0  Cognitive  Strategies 

-  Simpler  component  -  specific  intellectual  skills 

intellectual  skills  0  Attitudes 

°  Physical  skills  -  Intellectual  skills,  verbal 

-  Part  skills,  procedural  information 

rules 

0  Verbal  information 

-  Meaningfully  organized  sets  of  information 

To  demonstrate  how  objectives  in  the  various  domains  of  learning  interact 
with  each  other  in  reaching  the  final  outcomes  of  the  training  objectives, 
the  team  will  use  the  format  titled  "Instructional  Maps."  This  map  will 
show  the  teaching  sequence  and  interactions  for  specific  training 
objectives  in  the  domains  of  information,  intellectual  skills,  attitudes, 
cognitive  strategies  and  motor  skills  as  well  as  the  relations  of 
intermediate  objectives  within  a  single  training  objective.  This  itemization 
will  help  insure  that  all  relevant  content  and  skills  are  included  in  the 
training  program. 

To  insure  learning,  sufficient  practice  time  and  exercises  must  be 
available.  At  a  minimum,  one  practice  exercise  will  be  designed  for  each 
intermediate  and  training  objective.  To  insure  the  transfer  and 
integration  of  learning ,  the  practice  exercises  will  relate  directly  to  the 
required  job  skills. 

Means  to  evaluate  learner  achievement  within  each  of  the  domains  of 
learning  will  be  itemized.  The  training  objectives  for  the  POI  will  be 
used  as  a  basis  for  deriving  test  problems  and  exercises.  In  addition  to 
itemizing  the  questions  deemed  essential,  a  few  additional  questions  will 
be  designed  and  included  to  check  the  reliability  of  responses  and  to 
measure  the  influence  of  changes  in  wording  and  teaching  techniques. 
Two  or  more  roughly  equivalent  or  closely  related  questions  (well- 
separated  in  the  test)  will  be  included  in  order  to  measure  consistency 
(i.e.,  validity)  of  answers. 

(3. 3. 2.1,  3. 3. 2. 2,  3. 3. 2. 3,  3. 3. 2. 4) 

Each  lesson  plan  will  specify  the  instructional  strategies  to  be 
employed  to  meet  both  interim  and  training  objectives.  The  strategies  will 
be  keyed  to  the  lesson  plan  outline  and  serve  to  cue  the  instructor. 
Training  content  related  to  knowledge  will  present  strategies  to  guide 
instructor-learner  discussions.  Skill  related  training  (physical-mental) 
strategies  will  present  cues  to  guide  performance/practice  oriented 
training.  Attitude  training  cues  will  aid  the  instructor  in  presenting  role 
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plays,  demonstrations  or  discussions.  Each  lesson  plan  will  be  designed 
so  that  the  interim  objectives  are  sequenced  to  provide  a  learning 
hierarchy  to  meet  the  training  objective.  Through  performance  oriented 
training  the  learners  will  apply  the  instructional  content.  The  lesson 
plans  will  provide  cues  to  the  instructor  to  guide  each  phase  of 
instructions.  All  tasks,  conditions  and  standards  will  be  presented  as 
well  as  associated  instructional  strategies.  Each  training  objective  and 
associated  interim/intermediate  objectives  will  be  tested  against  the  stated 
task  conditions  and  performance  standards.  A  training  record,  keyed  to 
the  POI,  will  be  designed  as  an  aid  to  record  learner  achievement. 
Strategies  will  be  provided  on  each  lesson  plan  to  guide  remedial  training. 

(3. 3. 3.1,  3. 3. 3. 2,  3. 3. 3. 3,  3. 3. 3. 4) 

The  team  will  insure  that  the  administrative  instructions  on  each 
lesson  plan  include  the  training  facilities  required  to  support  the 
training,  equipment  needed  to  support  the  training,  instructional 
personnel  required  and  specifications  of  time.  A  training  course  guide 
will  be  developed  to  support  required  POI.  The  course  guide  will  contain 
the  following  four  sections: 

0  An  Introduction,  which  will  describe  the  Guide's  organization, 
list  the  program's  instructional  units  and  provide  an  overview 
of  the  training  administrator's  responsibilities. 

0  A  detailed  discussion  of  the  instructional  program,  including  its 
overall  objective(s) ,  specific  learner  performance  objectives  and 
the  contents  and  duration  of  each  instructional  unit. 

°  A  detailed  discussion  of  program  planning  considerations, 

including  scheduling,  desirable  class  size,  instructor 
qualifications,  materials  and  equipment  required,  facilities 
needed  and  procedures  for  estimating  program  costs. 

°  A  detailed  discussion  of  program  management  and  evaluation 

considerations,  including  suggestions  for  record  keeping  and 
identification  of  potential  improvements  to  the  program. 


Figure  B-10  (Concluded) 
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EXPLANATORY  NOTES 


Block 

Name 

Functional 

DescriDtion 

SIGN-ON 

Typical  sign-on  procedures  for  access¬ 
ing  an  interactive  computer  system 
and  specific  software  packages. 

Restricts  acbess  to  STM  to  those 
who  are  authorized,  i.e.,  those  who 
know  the  sign-on  code. 

DECISION/QUERY: 
does  operator  wish  to 
view  some  basic 
descriptive  info  about 
STM? 

For  the  novice  user,  a  capability  will 
exist  to  access  and  review  basic 
descriptive  information  about  STM, 
e.g.,  its  applications,  its  structure  and 
data  base,  its  interactive  procedures, 
a  list  of  more  detailed  references, 
etc.  If  the  user  so  chooses,  this 
information  can  be  called  up  on  the 
screen,  in  a  series  of  display  pages. 

DECISION/QUERY: 
does  operator  wish  to 
view  the  list  of  sys¬ 
tem  types  to  which 
STM  presently  is 
applicable? 

At  any  stage  of  its  evolution,  STM's 
data  base  will  contain  complete 
taxonomies  for  only  a  finite  (but 
steadily  growing)  set  of  system  types. 
Unless  the  user  is  interested  in  one 
of  the  types  presently  "stored"  in  the 
data  base,  STM  will  not  be  applicable 
to  the  user’s  needs.  If  the  user  so 
chooses,  the  list  of  presently  "stored" 
types  can  be  called  up  on  the  screen. 
For  the  proposed  demonstration,  that 
list  will  include  5  entries,  only  one  of 
which  (Training  Bystems)  actually  will 
have  taxonomies  in  storage. 

SYSTEM  TYPE 
SELECTION 

The  user  will  type  in  the  full  name  of 
one  of  the  system  types  for  which 
taxonomies  presently  exist  in  the  data 
base.  For  the  proposed  demonstration, 
even  though  four  other  choices  will 
"appear”  to  be  available,  the  user  will 
have  to  select  "Training". 

APPLICATION 

SELECTION 

The  computer  will  acknowledge  the 
user's  interest  in  the  selected  system 
type  (Training)  and  inquire  whether 
the  user  wishes  to  apply  STM  to  1) 
select  MOEs/MOPs  or  2)  specify  a  sys¬ 
tem  design.  Even  though  both  options 
will  be  displayed,  only  1)  ...  measures 
selection  ...  actually  will  be  available 
for  the  proposed  demonstration.  Thus, 
the  user  wUl  have  to  type  in  "1". 

EXPLANATORY  NOTES 


issii 

Block 

Number 

Block 

Name 

Functional 

DescriDtion 

2 

6 

SUBSYSTEM 

SELECTION 

The  computer  will  acknowledge  the 
user’s  interest  in  the  specified  STM 
application  (measures  selection),  and 
will  present  a  list  of  the  general  or 
typical  subsystems  of  the  selected  sys¬ 
tem  type  (Training).  The  computer 
then  will  ask  the  user  to  indicate 
which  subsystem  is  to  be  addressed 
first.  Even  though  all  six  general  sub¬ 
systems  of  Training  will  be  displayed 
to  the  user,  only  two  of  them 
(Curriculum  Development  and  Instructor 
Preparation)  actually  will  have  taxono¬ 
mies  in  storage  for  the  proposed  demo. 

2 

7 

DECISION/QUERY: 
does  operator  wish  to 
view  the  list  of  cur¬ 
rent  Performance 
aspect  KEYWORDS  for 
the  selected  subsys¬ 
tem? 

In  any  measurement  application,  the 
user  typically  is  interested  in  assessing 
how  well  the  system  performs  some 
particular  aspect  of  its  work.  For 
example,  one  might  be  preparing  to 
select  MOEs/MOPs  to  investigate  how 
well  a  curriculum  development  subsys¬ 
tem  has  performed  with  respect  to 
producing  specifications  for  testing 
learners.  Each  aspect  of  performance 
associates  with  some  particular  subset 
of  the  system's  complete  taxonomy. 

Each  taxon  in  the  subset  gives  rise  to 
certain  issues  and  implications  for 
assessing  the  associated  aspect  of  per¬ 
formance. 

At  any  stage  of  its  evolution,  STM's 
data  base  will  contain  specifications  of 
the  taxonomy  subsets  associated  with  a 
finite  list  of  performance  aspects  for 
any  given  system /subsystem.  Having 
signified  interest  in  Training"  systems 
and  (for  example)  their  "curriculum 
development"  or  "instructor  preparation" 
subsystem,  the  user  can  elect  to  call 
up  display  of  a  list  of  the  performance 
aspects  for  the  selected  subsystem  for 
which  subsets  of  taxa  are  defined  in 
the  data  base. 

2 

8 

PERFORMANCE 

ASPECT 

SPECIFICATION 

The  user  can  type  in  any  KEYWORD 
...  whether  or  not  it  exists  in  the  cur¬ 
rent  list  ...  to  signify  the  performance 
aspect  of  interest.  For  the  proposed 
demonstration,  the  KEYWORD  current 
lists  for  the  Curriculum  Development 

EXPLANATORY  NOTES 
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8 

(cont'd) 

and  Instructor  Preparation  subsystems 
will  be  relatively  short  (approx.  10 
entries  in  each).  Further,  actual  sub¬ 
sets  of  taxa  will  be  defined  in  the 
data  base  for  only  2  or  3  entries  in 
each  list.  One  entry  in  both  lists  will 
be  "TESTING",  and  taxa  subsets  will 
be  defined  for  that  entry. 

9 

COMPUTER 

DECISION:  is  the 
performance  aspect 
designated  by  the  user 
contained/defined  in 
the  current  KEYWORD 
list  for  the  selected 
subsystem? 

If  the  entered  KEYWORD  is  not  now 
defined  (in  terms  of  a  designated  sub¬ 
set  of  taxa),  the  computer  will  offer 
the  user  an  opportunity  to  review  the 
subsystem's  complete  taxonomy  in 
order  to  identify  those  taxa  associated 
with  the  newly  specified  performance 
aspect  of  interest.  This  is  one  exam¬ 
ple  of  how  the  applicability  of  STM 
can  grow  adaptively:  a  properly 
authorized  user  can  define  a  new  sub¬ 
set  of  the  taxonomy,  and  this  subset 
can  be  accessed  in  the  future  by  other 
users  interested  in  that  same  aspect  of 
performance. 

10 

DECISION/QUERY: 
does  the  operator  wish 
to  define  a  subset  of 
taxa  associated  with 
the  previously  unde¬ 
fined  KEYWORD  that 
has  been  entered? 

Upon  finding  that  the  entered  KEY¬ 
WORD  is  not  contained  in  the  current 
list  for  the  subsystem  of  interest 
(Curriculum  Development  or  Instructor 
Preparation),  the  computer  notifies  the 
user  that  an  undefined  aspect  of  per¬ 
formance  has  been  designated,  and 
inquires  whether  the  user  wishes  to 
specify  the  taxa  relevant  to  that 
aspect. 

If  the  user  elects  not  to  define  a 
taxonomy  subset,  nothing  more  can  be 
done  concerning  the  undefined  aspect 
of  performance.  STM  defaults  to  the 
user,  who  can  then  choose  to  specify 
another  aspect  of  performance,  or 
address  another  type  of  system 
entirely,  or  simply  commence  SIGN- 
OFF  procedures. 

11 

DESIGNATE  TAXA 
(sequential  procedure) 

Upon  receiving  an  indication  that  the 
user  intends  to  define  a  taxonomy  sub¬ 
set  associated  with  the  newly  specified 
aspect  of  performance,  the  computer 
begins  to  display  the  specified  subsys¬ 
tem's  taxonomy  to  the  user.  The 

Block 

Number  I  Number 


11 

(cont'd) 
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DECISION/QUERY: 
does  operator  wish  to 
record  for  permanent 
storage  the  new 
taxonomy  subset  just 
defined? 


ENTER 

AUTHORIZATION 

CODE 


concept  here  is  that  the  user  will  make 
an  "include  or  exclude"  decision  con¬ 
cerning  each  taxon's  membership  in  the 
new  subsystem.  Experience  suggests 
that  it  is  sufficient  to  examine  only 
the  characteristics  level  taxa.  If  a 
taxon  on  that  level  associates  with 
some  aspect  of  performance,  it  auto¬ 
matically  follows  that  its  hierarchically 
superior  taxa  on  the  functional  pur¬ 
poses  level  and  the  objectives  level 
also  must  associate  with  the  perfor¬ 
mance  aspect.  Thus,  for  example,  if 
the  user  decides  that  taxon  1.2.4.1 
(characteristics  level)  is  relevant  to  the 
aspect  of  interest  and  signifies  that  the 
taxon  is  to  belong  to  the  subsystem 
being  formed,  the  computer  automati¬ 
cally  will  include  taxa  1.2.4  (functional 
purposes  level)  and  1.2  (objectives 
level)  in  the  subsystem  as  well. 

Thus,  the  computer  will  display  every 
characteristics-level  taxon  to  the  opera¬ 
tor,  in  sequence,  and  will  Insist"  that 
the  user  signify  either  INCLUDE  or 
EXCLUDE  before  moving  on  to  the 
next  taxon.  Once  all  characteristics- 
level  taxa  have  been  reviewed  and 
designated  for  inclusion/exclusion,  a 
subset  will  have  been  defined  for  the 
new  performance  aspect. 

The  computer  will  not  permit  "casual" 
manipulation/modification  of  its  data 
base  of  taxonomies,  systems,  subsys¬ 
tems,  performance  aspect  subsets,  etc. 
Any  user  can,  at  any  time,  define  and 
operate  on  a  new  performance  aspect 
and  taxonomy  subset.  But  the  com¬ 
puter  will  not  permanently  store  those 
data  unless  proper  authorization  is 
received. 

If  the  user  desires  to  order  permanent 
storage  of  a  newly  defined  aspect  of 
performance  and  its  associated  tax¬ 
onomy  subset,  the  user  must  tjrpe  in  a 
special  authorization  code.  This  is  not 
the  same  as  the  SIGN-ON  code.  Only" 
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3 

13 

special,  designated  users  will  have  the 

(cont'd) 

authority  to  add  or  delete  data  in  the 
permanent  storage  bases. 

3 

14 

COMPUTER 

If  the  entered  code  is  valid,  the  com- 

DECISION:  is  the 

puter  will  enter  the  newly  defined 

code  entered  by  the 

performance  aspect  and  its  associated 

user  valid  authorize- 

taxonomy  subset  into  permanent  stor- 

tion  for  permanent 

age.  Future  users  then  will  be  able 

storage? 

to  access  and  use  the  subset.  If  the 
entered  code  is  not  valid,  no  perma¬ 
nent  storage  will  take  place,  but  the 
newly  defined  aspect  and  subset 
remains  available  for  use  by  this  user 
on  this  application. 

3 

15 

DISPLAY:  the  struc- 

The  computer  presents  the  user  with 

ture  and  size  of  the 

a  summary  of  the  structure  and  size 

taxonomy  subset 

of  the  taxonomy  subset  associated 
with  the  specified  aspect  of  perfor- 

mance.  This  will  entail  a  listing  of 
the  numbers  of  hierarchies  and  indi- 

vidual  taxa  in  each  STM  "column" 
associated  with  the  specified  perfor¬ 
mance  aspect.  A  hierarchy,  in  this 
context,  is  a  set  of  associated  taxa, 
headed  by  one  objectives-level  taxon 
and  followed  by  its  descendants  on 
the  two  lower  levels,  but  including 
only  those  descendants  that  are  rele¬ 
vant  to  the  performance  aspect  of 
interest.  For  example,  one  relevant 
hierarchy  might  consist  of: 

1.2 

1.2.1 

1.2. 1.3 

1. 2.1.4 

1.2.4 

1.2. 4. 2 

1.2. 4. 3 

1. 2.4.6 

That  particular  example  depicts  a 
hierarchy  of  performance  potentiality 
taxa  (leading  digit  =  1)  containing  8 
entries,  namely,  the  second  objectives- 
level  "patriarch"  (1.2),  its  first  func- 

tional  purposes-level  derivative  taxon 
(1.2.1),  that  taxon's  third  and  fourth 

characteristics-level  descendants 

c-iz 
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(1. 2.1.3,  1. 2.1.4),  the  "patriarch's" 
fourth  functional  purposes  taxon 
(1.2.4),  and  that  taxon's  second,  third, 
and  sixth  eharacteristics-level  deriva¬ 
tives  (1.2.4. 2,  1.2. 4. 3,  1.2. 4. 6). 

For  the  proposed  demonstration,  a 
typical  size/structure  message  dis¬ 
played  to  the  operator  might  appear  as 
follows: 

SYSTEM:  TRAINING 

SUBSYSTEM:  CURRICULUM 
DEVELOPMENT 
ASPECT:  TESTING 


POTENTIALITIES: 

PROCESSES: 

PRODUCTS: 


5  hierarchies  42  taxa 
2  hierarchies  10  taxa 

6  hierarchies  44  taxa 


This  information  conveys  a  summary  of  the 
scope  of  the  measurement  application 
at  hand,  and  provides  a  "feel"  for  the 
distribution  of  measurement  issues 
among  potentialities,  processes,  and 
products. 

The  user  may  wish  to  examine  the  hier¬ 
archies  and  their  constituent  taxa.  The 
STM  data  base  "claims"  that  those  hier¬ 
archies  are  relevant,  in  general,  to  the 
kind  of  measurement  application  at 
hand.  However,  the  user  might  decide 
that  certain  taxa,  or  even  entire  hier¬ 
archies,  are  not  really  pertinent  to  the 
particular  application  and  the  particular 
system  with  which  the  user  is  concerned. 
Also,  the  user  may  have  reason  to 
believe  that  other  taxa  deemed  not 
generally  relevant  are  in  fact  of  impor¬ 
tance  in  this  particular  case.  Thus, 
each  user  needs  to  have  the  ability  to 
review  and  edit  (through  deletion  or 
addition)  the  taxonomy  subset  generally 
associated  with  the  performance  aspect 
to  insure  that  the  subset  is  "tailored" 
to  the  user's  particular  application. 
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CONTEXTUAL 

CATEGORY 

SELECTION 


REVIEW  (AND  EDIT) 
HIERARCHIES 
(sequential  procedure) 


DECISION/QUERY: 
does  the  user  wish  to 
review/edit  the  hier¬ 
archies  associated  with 
another  contextual 
category? 

PERMANENT 
RECORDING  OPTION 
ROUTINE 


If  the  user  signifies  a  desire  to  review 
and  possibly  edit  the  taxonomy  subset, 
the  computer  inquires  as  to  which  con¬ 
textual  category  of  taxa  the  user 
wishes  to  review  first.  The  choices 
correspond  to  the  five  columns  of  STM. 
For  the  proposed  demonstration,  even 
though  all  five  options  will  "appear"  to 
be  available,  no  taxa  actually  will  be 
in  storage  for  the  "Environment 
Specification”  or  "General  Constraints" 
categories.  Thus,  the  demonstration 
user  will  be  restricted  to  choosing  one 
of  the  three  "Performance"  categories 
(Potentialities,  Processes,  or  Products). 

Once  the  user  indicates  which  contex¬ 
tual  category  of  taxa  is  desired,  the 
computer  begins  to  display  the  hier¬ 
archies  of  the  subset  that  belong  to 
that  category.  At  each  taxon,  the 
user  will  be  able  to  command  a  dele¬ 
tion.  Such  action  will  drop  the  indi¬ 
cated  taxon  (and  its  associated 
measures  and  measurement  implica¬ 
tions)  from  the  set  of  taxa  relevant 
to  the  particular  measurement  applica¬ 
tion.  The  user  also  will  have  the 
option  of  commanding  an  addition,  i.e., 
the  option  of  defining  a  new  taxon  and 
including  it  in  the  set  of  taxa  for  the 
particular  application.  The  user  will 
review  the  entire  collection  of  hier¬ 
archies  in  this  fashion,  indicating  (at 
each  taxon)  whether  to  retain,  delete, 
augment,  or  replace  the  various  taxa. 
Thus,  the  hierarchies  will  be  "tailored" 
to  the  user's  particular  application. 

Having  completed  the  review/editing  of 
hierarchies  in  one  column  of  the  STM 
(e.g.,  Potentialities),  the  user  will  have 
the  option  of  pierforming  the  same 
function  for  the  hierarchies  of  another 
column  (e.g.,  Processes  or  Products). 

Certain  specially  authorized  users  will 
have  the  capability  of  making  perma¬ 
nent  editorial  changes  to  the  data 
bases  of  hierarchies.  This  will  be  done 
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via  entry  of  a  special  code  number,  in 
a  fashion  similar  to  that  described  in 
blocks  12,  13,  and  14. 
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DECISION/QUERY: 
does  operator  wish  to 
review,  and  possibly 
edit,  the  measurement 
issues  associated  with 
the  relevant  taxa? 


Up  to  this  point,  the  user  will  have 
1)  specified  the  system  and  subsystem 
of  interest,  2)  defined  the  measure¬ 
ment  application  in  terms  of  the  spe¬ 
cific  aspect  of  performance  of  interest 
and  3)  selected  the  hierarchies  of 
taxa  that  are  relevant  to  that  meas¬ 
urement  application.  The  next  step  is 
to  examine  precisely  what  those  taxa 
portend  for  that  measurement  applica¬ 
tion,  i.e.,  the  issues  and  implications 
for  measurement  associated  with  each 
taxon.  Of  course,  if  a  particular 
taxon  has  been  newly  defined  by  the 
user,  the  computer  will  as  yet  have 
no  associated  issues  for  that  taxon  in 
storage.  Thus,  the  user  may  wish  to 
define  such  issues,  and  place  them  in 
storage. 


4 


22 


DESCRIPTION 
LEVEL  SELECTION 


If  the  user  signifies  a  desire  to  review 
and  possibly  edit  the  measurement 
issues  associated  with  the  selected 
taxa,  the  computer  inquires  as  to 
which  level  of  system  description 
(objectives,  functional  purposes,  or 
characteristics)  the  user  wishes  to 
review  first.  Even  though  all  three 
choices  will  "appear"  to  be  available, 
the  proposed  demonstration  will  be 
limited  to  only  the  Objectives  and 
Functional  Purposes  levels. 


4 
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REVIEW  (AND  EDIT) 
ISSUES  SET'S 
(sequential  procedure) 


Once  the  user  indicates  which  level 
of  system  description  is  desired,  the 
computer  begins  to  display  the  taxa 
of  that  level,  along  with  the  measure¬ 
ment  issues  and  implications  associated 
with  each  taxon.  The  user  has  the 
opportunity  to  edit  those  issues  and 
implications,  by  deletion,  modification, 
or  addition.  Thus,  the  user  can  select 
precisely  which  issues  will  be  addressed 
in  the  measurement  application  at 
hand. 
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4 

24 

DECISION/QUERY: 
does  the  user  wish  to 
review/edit  the  meas¬ 
urement  issues  associ¬ 
ated  with  another 
level  of  system  des¬ 
cription? 

Having  completed  the  review/editing  of 
measurement  issues  in  one  row  of  the 
STM  (e.g.,  Objectives  Level),  the  user 
will  have  the  opportunity  to  perform 
the  same  process  for  the  issues  of 
another  row  (e.g.,  Functional  Purposes 
level  or,  except  for  the  proposed 
demonstration.  Characteristics  level). 

4 

25 

PERMANENT 
RECORDING  OPTION 
ROUTINE 

Certain  specially  authorized  users  will 
have  the  capability  of  making  perma¬ 
nent  editorial  changes  to  the  data 
bases  of  measurement  issues.  This 
will  be  done  via  entry  of  a  special 
code  number,  in  a  fashion  similar  t 
that  described  in  Mocks  12,  13,  ana  14. 

4 

26 

DECISION /QUERY: 
does  the  user  wish  to 
review,  and  possibly 
edit,  measures 
applicable  to  selected 
issues? 

This  is  the  final,  and  most  important, 
step  in  the  STM  application.  It  is  at 
this  point  that  a  user  extracts  and/or 
defines  specific  measures  suited  to 
the  performance  aspect  of  interest. 

By  signifying  a  desire  to  review  the 
applicable  measures,  the  user  informs 
the  computer  that  access  to  the  data 
bases  of  measures  is  desired. 

4 

27 

INDIVIDUAL  TAXON 
SPECIFICATION 

The  computer  responds  by  requesting 
the  user  to  designate  a  relevant  taxon 
from  the  (previously  identified  and— 
possibly— edited)  subset.  The  user 
types  in  the  taxon's  number  (two-to- 
four  digits,  depending  on  the  level  of 
system  description).  For  the  proposed 
demonstration,  no  measures  will  actu¬ 
ally  be  in  storage  for  taxa  on  the 
characteristics  level  of  system  descrip¬ 
tion.  Thus,  the  demonstration  user 
will  be  restricted  to  entering  two-  or 
three-digit  taxa  numbers. 

4 

28 

MEASURES  SET 
DISPLAY 

Once  the  user  designates  a  taxon  of 
interest,  the  computer  responds  by  dis¬ 
playing  all  of  the  relevant  measure¬ 
ment  issues  associated  with  that  taxon, 
along  with  specific  measures  suitable 
for  addressing  those  issues. 
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4 

29 

REVIEW  (AND  EDIT) 
MEASURES 

The  user  has  the  opportunity  to  edit 
the  measures  set  associated  with  the 
sp  ?>fied  taxon,  through  deletion,  modi¬ 
fication,  or  addition.  Thus,  the  spe¬ 
cific  measures  selected  can  be 
"tailored"  to  the  particular  measure¬ 
ment  application. 

4 

30 

DECISION/QUERY: 
does  the  user  wish  to 
review/edit  the  meas¬ 
ures  set  associated 
with  another  taxon? 

Having  completed  the  review/editing  of 
measures  associated  with  one  member 
of  the  taxa  subset,  the  user  will  have 
the  opportunity  to  perform  the  same 
process  for  another  taxon's  measures. 
Again,  for  the  proposed  demonstration, 
only  Objectives  level  and  Functional 
Purposes  level  taxa  actually  will  have 
measures  sets  available  for  review. 

4 

31 

PERMANENT 
RECORDING  OPTION 
ROUTINE 

Certain  specially  authorized  users  will 
have  the  capability  of  making  perma¬ 
nent  editorial  changes  to  the  data 
bases  of  measures.  Ibis  will  be  done 
via  entry  of  a  special  code  number,  in 
a  fashion  similar  to  that  described  in 
blocks  12,  13,  and  14. 

5 

32 

DECISION/QUERY: 
does  the  user  wish  to 
address  another  aspect 
of  performance  for 
this  same  subsystem? 

Once  the  user  has  reviewed/edited  all 
measures  of  interest  to  the  perfor¬ 
mance  aspect  previously  specified,  the 
computer  inquires  whether  the  user 
wishes  to  consider  some  other  perfor¬ 
mance  aspect  as  well.  An  affirmative 
response  by  the  user  causes  STM  to 
branch  back  to  block  7  on  page  2. 

5 

33 

DECISION/QUERY: 
does  the  user  now 
wish  to  address  some 
other  subsystem  of 
this  same  system? 

If  the  user  indicates  that  no  more 
work  remains  to  be  done  in  the  con¬ 
text  of  the  originally  specified  sub¬ 
system,  the  computer  inquires  whether 
the  user  wishes  to  consider  another 
subsystem  as  well.  An  affirmative 
response  by  the  user  causes  STM  to 
branch  back  to  block  6  on  page  2. 

5 

34 

DECISION/QUERY: 
does  the  user  now 
wish  to  address  some 
other  system  entirely? 

If  the  user  indicates  that  no  more 
work  remains  to  be  done  with  any  of 
the  subsystems  of  the  originally  spe¬ 
cified  system,  the  computer  inquires 
whether  the  user  wishes  to  consider 
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another  system  as  well.  An  affirma¬ 
tive  response  by  the  user  causes  STM 
to  branch  back  to  block  3  on  page  1. 

Typical  sign-off  procedures  for  ter¬ 
minating  work  with  an  interactive 
computer  system.  Permits  storage  of 
data  processing  in  user-controlled 
memory  files,  initiates  cost-accounting 
processing,  etc. 


